Skip to content

feat(hang,mux,hls): rework the timeline as a single track of complete segments - #2547

Draft
kixelated wants to merge 4 commits into
mainfrom
claude/timeline-segments-rework-w6yeew
Draft

feat(hang,mux,hls): rework the timeline as a single track of complete segments#2547
kixelated wants to merge 4 commits into
mainfrom
claude/timeline-segments-rework-w6yeew

Conversation

@kixelated

@kixelated kixelated commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

The broadcast now has one timeline track (timeline.z, advertised at the catalog root) carrying one self-contained record per aligned segment. This is the second iteration of the design, replacing the per-track timelines + cadence inference from the first push (see the commit history for the intermediate design).

The record:

{
  "segment": 42,
  "pts": 123456000,
  "duration": 6000000,
  "tracks": {
    "video": [{ "start": 105, "end": 105 }],
    "audio": [{ "start": 512, "end": 540 }, { "start": 550, "end": 560 }]
  }
}
  • Self-contained records: explicit duration (live edge usable immediately; next.pts != pts + duration marks a discontinuity) and per-track arrays of inclusive group ranges, so serving seg/{n}.m4s needs only record N. Multiple ranges represent sequence gaps (Elemental-style); an absent track is a whole-segment gap (EXT-X-GAP); an optional keyframe: false per range flags a non-IDR range start.
  • Declared durationMax: the catalog's timeline section carries a required durationMax (in timescale units), because the boundary owner knows its segment duration up front: the encoder its keyframe cadence, the HLS import its source playlist's EXT-X-TARGETDURATION (which it declares before the first init segment). The bound is fixed at Segmenter construction (or set_duration_max until the timeline track exists; frozen once advertised) and the HLS export writes EXT-X-TARGETDURATION straight from it, with no observed-duration guessing.
  • Facts up, policy down (moq_mux::timeline::Segmenter): media tracks enroll (Segmenter::track(name, kind)) and report every group open; they never decide boundaries. Boundaries come from Segmenter::cut(pts) when the application knows them, or auto-cut on the driver track's keyframes (first video, else first track). Auto-cut enforces the declared bound: when the next driver keyframe would overrun durationMax, the segment closes retroactively at the previous keyframe (which also puts the boundary on a keyframe), so only a single GOP longer than the bound yields a longer segment, and that is logged once. The first explicit cut disables auto-cut for good. No wall clocks anywhere, timestamps only.
  • Completeness-based flush: a record publishes only once every enrolled track reported a group at or past the segment's end boundary (or closed). Cuts registered ahead of the media (HLS import) just wait for it. A dead track's recorder dropping is what unblocks its segments; catalog ownership makes that explicit.
  • Explicit cuts wired end to end: the fMP4 importer cuts on styp atoms (CMAF segments on disk; with_cut_on_boundary(false) to opt out) and exposes cut(); the HLS import cuts per source-playlist segment on the primary variant only, so importing then exporting reproduces the origin's segmentation (including its target duration).
  • Wire format: a moq-json stream (single group, one DEFLATE-compressed JSON record per frame, shared window), so verbose records compress against all earlier ones. Live history is bounded by the group cache; VOD reads a recording.
  • moq-hls export is now a pure function of the timeline: one timeline subscription fans records out to per-rendition windows; playlists agree on EXT-X-MEDIA-SEQUENCE/EXTINF by construction, render EXT-X-GAP (protocol version bumps to 8 only when used) and EXT-X-DISCONTINUITY, and take EXT-X-TARGETDURATION from the catalog. The old live-edge/final-duration estimation, target-duration latching, and dropped-counter bookkeeping are gone.
  • Spec: drafts/draft-lcurley-moq-hang.md gains a Timeline section (catalog section incl. durationMax, framing, record schema, segmentation rules incl. the bound-enforcement rule) and the catalog root schema now lists timeline.

Public API changes

All breaking; this PR was born as a breaking rework of an unspecced surface.

  • hang::timeline: Record is now {segment, pts, duration, tracks} + new Range; track_name(rendition) replaced by DEFAULT_NAME ("timeline.z"). hang::Catalog gains a root timeline section (catalog::Timeline::new(track, duration_max), required durationMax on the wire); VideoConfig/AudioConfig lose theirs.
  • moq_mux::timeline: Segmenter::new(duration_max) declares the bound (plus set_duration_max, which errors once the timeline is advertised; Default keeps the 2s convention); Cadence replaced by Kind {Video, Audio}; Recorder reports (sequence, pts, keyframe) and closes on drop; Producer is now the broadcast timeline (new(broadcast, &segmenter)); Entry carries duration: std::time::Duration and the ranges map; DEFAULT_INTERVAL/with_interval/with_duration_max replaced by DEFAULT_DURATION_MAX.
  • moq_mux::catalog::Producer: timeline() takes no name (the one broadcast timeline, advertised at the root on first use); media_producer(track, container, kind); new set_wall(pts, wall) that re-advertises the section (the timeline producer's own set_wall doesn't reach an already-published catalog).
  • moq_mux::container::fmp4::Import: new cut() and with_cut_on_boundary(bool).
  • moq_hls::export::segments::Segment drops group; gap/discontinuity surface through the playlist instead.
  • JS @moq/hang: Record/Range reshaped; Segmenter with track()/cut()/completeness flush, constructed with durationMax and exposing it for the catalog section; new Recorder (now the timeline prop of the legacy container producer); catalog timeline moves to the root schema with required durationMax.

These are breaking pub changes in core libraries, which per Branch Targeting would go to dev, but no dev branch exists on the remote, so this targets main. Happy to retarget.

Open questions

  • The auto-cut driver is elected (first video, else first track) rather than app-designated; cut() is the escape hatch for anything unusual (two independent video tracks, on-demand encoding).
  • A file import that cuts on styp (CMAF on disk) has no upfront source of durationMax, so it declares the 2s default; a file with longer on-disk segments gets a one-time warning that it overran the declared bound. A --segment-duration-style knob (or a deferred advertisement) could follow if that bites in practice.
  • The final segment's duration runs to the newest report, so it undercounts the last group's tail by up to one group duration.

Cross-package sync

  • js/hang mirrors the record shape, segmenter (including bound enforcement), and root catalog section.
  • doc/bin/hls.md updated (single timeline, gaps/discontinuities, TARGETDURATION from the declared bound, import preserving source segmentation).
  • drafts/draft-lcurley-moq-hang.md specs the timeline (validated with kramdown-rfc locally; nix isn't available in this sandbox so just drafts check proper runs in CI).
  • moq-ffi and the language wrappers do not expose the timeline; nothing to mirror.

Test plan

  • moq-mux segmenter tests: auto-cut pacing, bound enforcement with misaligned keyframes (segments close retroactively at the previous keyframe), honest overrun for GOPs longer than the bound, frozen-once-advertised durationMax, completeness gating, explicit cuts (multi-GOP segments), pre-registered cuts, audio-only pacing, late video takeover (now landing boundaries on the new driver's keyframes), startup race, sequence-gap ranges, whole-segment gaps, keyframe: false, closed-track unblocking, pre-attach buffering.
  • moq-hls: export end-to-end tests (playlists from the shared timeline, TARGETDURATION from the declared bound, aligned sequences, gap rows, recorder cursors draining after Broadcaster drop); fmp4 import test asserts the root section + both renditions indexed from group 0.
  • JS: bun tests mirror the Rust segmenter cases, including the new bound-enforcement ones.
  • cargo test green on hang, moq-mux, moq-hls, moq-json, moq-audio, moq-rtc (one pre-existing sandbox-environmental failure each in libmoq/moq-rtc, also failing on the base commit); cargo check --workspace --all-targets, cargo fmt --check, cargo clippy (0 warnings), cargo doc -D warnings on the touched crates, biome + tsc across all JS packages all pass.

(Written by Claude Fable)

Review fixes

An adversarial review found four defects in how records are timed and gated (not in the wire format). All fixed, each with a regression test verified to fail without the fix.

  • The final segment's duration was zero. It ran to the newest group start, so a final segment holding one GOP got duration: 0 and HLS rendered EXTINF:0. (This was listed as an open question below; it is worse than an undercount.) Tracks now report where their content ends as well as where each group opens: container::Producer carries max(timestamp + duration) and reports it on every cut, and the fMP4 passthrough path reports each fragment's end. New Recorder::end(pts), which advances the same frontier completeness is judged against.
  • EXT-X-TARGETDURATION could be smaller than an EXTINF. The declared durationMax is a target, not a guarantee: a single GOP longer than it cannot be split, so the segmenter publishes an honest long segment (and warns). The export still starts from the declared bound (so the first playlist carries the real value) but raises it to cover the window, since a playlist whose EXTINF exceeds TARGETDURATION is invalid.
  • The HLS import published records before its sibling renditions enrolled. Renditions are ingested one at a time and a record is immutable, judged against the tracks enrolled at the moment it flushes. The primary variant could publish several records before the others loaded their init segments, marking them EXT-X-GAP permanently and folding their first groups into whichever segment flushed next. New Segmenter::hold() (RAII, nests) defers flushing; the import holds one across each pass.
  • One dead rendition froze every healthy one. A rendition that kept failing stayed enrolled with a frozen frontier, and completeness gates on every enrolled track, so the whole broadcast's timeline stopped. Three consecutive failures now drop its importer, closing its recorders until it recovers.

Additional public API: moq_mux::timeline::{Hold, Segmenter::hold}; JS Segmenter.hold() (returns a dispose function) and Recorder.end(pts). The draft gains the rule that a consumer deriving a hard limit from durationMax must raise it to cover an overrunning record, and that the final segment's duration should carry the last group's content end.

Not changed: the timeline wire format is replaced outright with no versioning or migration, which the review also flagged. Nothing consumes the timeline yet, so there is nothing to stay compatible with.

(Written by Opus 5)

claude added 2 commits July 28, 2026 00:37
Timeline records now carry an aligned segment number: segment N covers
the same span of content time on every track of a broadcast, which is
what HLS requires of switchable renditions. A video segment is exactly
one group (opened by a keyframe); an audio segment packs every audio
group inside the same span.

A shared Segmenter per broadcast numbers the segments: the video
recorder (Cadence::Boundary) opens one per group, audio recorders
(Cadence::Aligned) record the first group at or after each boundary,
falling back to interval pacing when no video track exists. The HLS
export keys its playlist window, seg/{segment}.m4s URIs, and
EXT-X-MEDIA-SEQUENCE on the aligned numbers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
Replaces the per-track timeline design wholesale. The segmenter now
follows facts-up/policy-down: tracks enroll and report group opens,
boundaries come from explicit cut() calls (HLS import per source
segment, fMP4 styp atoms) or catalog-owned auto-cut on the driver
track's keyframes, and a record flushes only once every enrolled track
reported past the segment's end. Records are self-contained
({segment, pts, duration, tracks: {name: [{start, end, keyframe?}]}}),
published on one timeline.z track advertised at the catalog root, and
specced in draft-lcurley-moq-hang.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
@kixelated kixelated changed the title feat(hang,mux,hls): rework the timeline around aligned segments feat(hang,mux,hls): rework the timeline as a single track of complete segments Jul 28, 2026
claude and others added 2 commits July 28, 2026 04:21
The encoder (or cutter) knows its segment duration up front, so the
timeline's catalog section now carries a required durationMax and the
HLS export writes EXT-X-TARGETDURATION straight from it, dropping the
latched longest-observed-duration guess. The Segmenter takes the bound
at construction (or set_duration_max before the timeline track exists;
it is frozen once advertised), and auto-cut now enforces it: when the
next driver keyframe would overrun the bound, the segment closes
retroactively at the previous keyframe, so only a GOP longer than the
bound produces a longer segment (logged once). The HLS import declares
the source playlist's target duration before its first init segment.
Also routes the wall anchor through catalog::Producer::set_wall so the
advertised section is republished rather than silently stale.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
…d tracks

Four defects an adversarial review of the timeline rework surfaced, all in
how records are timed or gated rather than in the wire format.

A record's terminal duration was computed from the newest group *start*, so a
final segment containing a single GOP got duration 0 and HLS rendered
EXTINF:0. Tracks now report where their content ends as well as where each
group opens: container::Producer carries max(timestamp + duration) and reports
it on every cut, and the fMP4 passthrough path reports each fragment's end.

EXT-X-TARGETDURATION came straight from the declared durationMax, but the
bound is a target rather than a guarantee: a GOP longer than it cannot be
split, so the segmenter publishes an honest long segment. The export now
raises the target duration to cover the window, since a playlist with an
EXTINF above TARGETDURATION is invalid.

The HLS import ingests renditions one at a time, and a record is immutable and
judged against the tracks enrolled the moment it flushes. The primary variant
could therefore publish several records before its siblings loaded their init
segments, marking them EXT-X-GAP for good and folding their first groups into
whichever segment flushed next. Segmenter::hold() defers flushing, and the
import holds one across each pass.

Finally, a rendition that kept failing stayed enrolled with a frozen frontier,
which gates completeness for the whole broadcast: one dead variant froze every
healthy variant's playlist. Three consecutive failures now drop its importer,
closing its recorders until it recovers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants