feat(hang,mux,hls): rework the timeline as a single track of complete segments - #2547
Draft
kixelated wants to merge 4 commits into
Draft
feat(hang,mux,hls): rework the timeline as a single track of complete segments#2547kixelated wants to merge 4 commits into
kixelated wants to merge 4 commits into
Conversation
Timeline records now carry an aligned segment number: segment N covers
the same span of content time on every track of a broadcast, which is
what HLS requires of switchable renditions. A video segment is exactly
one group (opened by a keyframe); an audio segment packs every audio
group inside the same span.
A shared Segmenter per broadcast numbers the segments: the video
recorder (Cadence::Boundary) opens one per group, audio recorders
(Cadence::Aligned) record the first group at or after each boundary,
falling back to interval pacing when no video track exists. The HLS
export keys its playlist window, seg/{segment}.m4s URIs, and
EXT-X-MEDIA-SEQUENCE on the aligned numbers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
Replaces the per-track timeline design wholesale. The segmenter now
follows facts-up/policy-down: tracks enroll and report group opens,
boundaries come from explicit cut() calls (HLS import per source
segment, fMP4 styp atoms) or catalog-owned auto-cut on the driver
track's keyframes, and a record flushes only once every enrolled track
reported past the segment's end. Records are self-contained
({segment, pts, duration, tracks: {name: [{start, end, keyframe?}]}}),
published on one timeline.z track advertised at the catalog root, and
specced in draft-lcurley-moq-hang.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
The encoder (or cutter) knows its segment duration up front, so the timeline's catalog section now carries a required durationMax and the HLS export writes EXT-X-TARGETDURATION straight from it, dropping the latched longest-observed-duration guess. The Segmenter takes the bound at construction (or set_duration_max before the timeline track exists; it is frozen once advertised), and auto-cut now enforces it: when the next driver keyframe would overrun the bound, the segment closes retroactively at the previous keyframe, so only a GOP longer than the bound produces a longer segment (logged once). The HLS import declares the source playlist's target duration before its first init segment. Also routes the wall anchor through catalog::Producer::set_wall so the advertised section is republished rather than silently stale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XnWWQZv6hruCwj6uLkAf7K
…d tracks Four defects an adversarial review of the timeline rework surfaced, all in how records are timed or gated rather than in the wire format. A record's terminal duration was computed from the newest group *start*, so a final segment containing a single GOP got duration 0 and HLS rendered EXTINF:0. Tracks now report where their content ends as well as where each group opens: container::Producer carries max(timestamp + duration) and reports it on every cut, and the fMP4 passthrough path reports each fragment's end. EXT-X-TARGETDURATION came straight from the declared durationMax, but the bound is a target rather than a guarantee: a GOP longer than it cannot be split, so the segmenter publishes an honest long segment. The export now raises the target duration to cover the window, since a playlist with an EXTINF above TARGETDURATION is invalid. The HLS import ingests renditions one at a time, and a record is immutable and judged against the tracks enrolled the moment it flushes. The primary variant could therefore publish several records before its siblings loaded their init segments, marking them EXT-X-GAP for good and folding their first groups into whichever segment flushed next. Segmenter::hold() defers flushing, and the import holds one across each pass. Finally, a rendition that kept failing stayed enrolled with a frozen frontier, which gates completeness for the whole broadcast: one dead variant froze every healthy variant's playlist. Three consecutive failures now drop its importer, closing its recorders until it recovers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The broadcast now has one timeline track (
timeline.z, advertised at the catalog root) carrying one self-contained record per aligned segment. This is the second iteration of the design, replacing the per-track timelines + cadence inference from the first push (see the commit history for the intermediate design).The record:
{ "segment": 42, "pts": 123456000, "duration": 6000000, "tracks": { "video": [{ "start": 105, "end": 105 }], "audio": [{ "start": 512, "end": 540 }, { "start": 550, "end": 560 }] } }duration(live edge usable immediately;next.pts != pts + durationmarks a discontinuity) and per-track arrays of inclusive group ranges, so servingseg/{n}.m4sneeds only record N. Multiple ranges represent sequence gaps (Elemental-style); an absent track is a whole-segment gap (EXT-X-GAP); an optionalkeyframe: falseper range flags a non-IDR range start.durationMax: the catalog's timeline section carries a requireddurationMax(in timescale units), because the boundary owner knows its segment duration up front: the encoder its keyframe cadence, the HLS import its source playlist'sEXT-X-TARGETDURATION(which it declares before the first init segment). The bound is fixed atSegmenterconstruction (orset_duration_maxuntil the timeline track exists; frozen once advertised) and the HLS export writesEXT-X-TARGETDURATIONstraight from it, with no observed-duration guessing.moq_mux::timeline::Segmenter): media tracks enroll (Segmenter::track(name, kind)) and report every group open; they never decide boundaries. Boundaries come fromSegmenter::cut(pts)when the application knows them, or auto-cut on the driver track's keyframes (first video, else first track). Auto-cut enforces the declared bound: when the next driver keyframe would overrundurationMax, the segment closes retroactively at the previous keyframe (which also puts the boundary on a keyframe), so only a single GOP longer than the bound yields a longer segment, and that is logged once. The first explicit cut disables auto-cut for good. No wall clocks anywhere, timestamps only.stypatoms (CMAF segments on disk;with_cut_on_boundary(false)to opt out) and exposescut(); the HLS import cuts per source-playlist segment on the primary variant only, so importing then exporting reproduces the origin's segmentation (including its target duration).moq-jsonstream (single group, one DEFLATE-compressed JSON record per frame, shared window), so verbose records compress against all earlier ones. Live history is bounded by the group cache; VOD reads a recording.EXT-X-MEDIA-SEQUENCE/EXTINFby construction, renderEXT-X-GAP(protocol version bumps to 8 only when used) andEXT-X-DISCONTINUITY, and takeEXT-X-TARGETDURATIONfrom the catalog. The old live-edge/final-duration estimation, target-duration latching, and dropped-counter bookkeeping are gone.drafts/draft-lcurley-moq-hang.mdgains a Timeline section (catalog section incl.durationMax, framing, record schema, segmentation rules incl. the bound-enforcement rule) and the catalog root schema now liststimeline.Public API changes
All breaking; this PR was born as a breaking rework of an unspecced surface.
hang::timeline:Recordis now{segment, pts, duration, tracks}+ newRange;track_name(rendition)replaced byDEFAULT_NAME("timeline.z").hang::Cataloggains a roottimelinesection (catalog::Timeline::new(track, duration_max), requireddurationMaxon the wire);VideoConfig/AudioConfiglose theirs.moq_mux::timeline:Segmenter::new(duration_max)declares the bound (plusset_duration_max, which errors once the timeline is advertised;Defaultkeeps the 2s convention);Cadencereplaced byKind {Video, Audio};Recorderreports(sequence, pts, keyframe)and closes on drop;Produceris now the broadcast timeline (new(broadcast, &segmenter));Entrycarriesduration: std::time::Durationand the ranges map;DEFAULT_INTERVAL/with_interval/with_duration_maxreplaced byDEFAULT_DURATION_MAX.moq_mux::catalog::Producer:timeline()takes no name (the one broadcast timeline, advertised at the root on first use);media_producer(track, container, kind); newset_wall(pts, wall)that re-advertises the section (the timeline producer's ownset_walldoesn't reach an already-published catalog).moq_mux::container::fmp4::Import: newcut()andwith_cut_on_boundary(bool).moq_hls::export::segments::Segmentdropsgroup; gap/discontinuity surface through the playlist instead.@moq/hang:Record/Rangereshaped;Segmenterwithtrack()/cut()/completeness flush, constructed withdurationMaxand exposing it for the catalog section; newRecorder(now thetimelineprop of the legacy container producer); catalogtimelinemoves to the root schema with requireddurationMax.These are breaking
pubchanges in core libraries, which per Branch Targeting would go todev, but nodevbranch exists on the remote, so this targetsmain. Happy to retarget.Open questions
cut()is the escape hatch for anything unusual (two independent video tracks, on-demand encoding).styp(CMAF on disk) has no upfront source ofdurationMax, so it declares the 2s default; a file with longer on-disk segments gets a one-time warning that it overran the declared bound. A--segment-duration-style knob (or a deferred advertisement) could follow if that bites in practice.durationruns to the newest report, so it undercounts the last group's tail by up to one group duration.Cross-package sync
js/hangmirrors the record shape, segmenter (including bound enforcement), and root catalog section.doc/bin/hls.mdupdated (single timeline, gaps/discontinuities, TARGETDURATION from the declared bound, import preserving source segmentation).drafts/draft-lcurley-moq-hang.mdspecs the timeline (validated with kramdown-rfc locally; nix isn't available in this sandbox sojust drafts checkproper runs in CI).moq-ffiand the language wrappers do not expose the timeline; nothing to mirror.Test plan
moq-muxsegmenter tests: auto-cut pacing, bound enforcement with misaligned keyframes (segments close retroactively at the previous keyframe), honest overrun for GOPs longer than the bound, frozen-once-advertiseddurationMax, completeness gating, explicit cuts (multi-GOP segments), pre-registered cuts, audio-only pacing, late video takeover (now landing boundaries on the new driver's keyframes), startup race, sequence-gap ranges, whole-segment gaps,keyframe: false, closed-track unblocking, pre-attach buffering.moq-hls: export end-to-end tests (playlists from the shared timeline, TARGETDURATION from the declared bound, aligned sequences, gap rows, recorder cursors draining afterBroadcasterdrop); fmp4 import test asserts the root section + both renditions indexed from group 0.cargo testgreen onhang,moq-mux,moq-hls,moq-json,moq-audio,moq-rtc(one pre-existing sandbox-environmental failure each inlibmoq/moq-rtc, also failing on the base commit);cargo check --workspace --all-targets,cargo fmt --check,cargo clippy(0 warnings),cargo doc -D warningson the touched crates, biome +tscacross all JS packages all pass.(Written by Claude Fable)
Review fixes
An adversarial review found four defects in how records are timed and gated (not in the wire format). All fixed, each with a regression test verified to fail without the fix.
duration: 0and HLS renderedEXTINF:0. (This was listed as an open question below; it is worse than an undercount.) Tracks now report where their content ends as well as where each group opens:container::Producercarriesmax(timestamp + duration)and reports it on everycut, and the fMP4 passthrough path reports each fragment's end. NewRecorder::end(pts), which advances the same frontier completeness is judged against.EXT-X-TARGETDURATIONcould be smaller than anEXTINF. The declareddurationMaxis a target, not a guarantee: a single GOP longer than it cannot be split, so the segmenter publishes an honest long segment (and warns). The export still starts from the declared bound (so the first playlist carries the real value) but raises it to cover the window, since a playlist whoseEXTINFexceedsTARGETDURATIONis invalid.EXT-X-GAPpermanently and folding their first groups into whichever segment flushed next. NewSegmenter::hold()(RAII, nests) defers flushing; the import holds one across each pass.Additional public API:
moq_mux::timeline::{Hold, Segmenter::hold}; JSSegmenter.hold()(returns a dispose function) andRecorder.end(pts). The draft gains the rule that a consumer deriving a hard limit fromdurationMaxmust raise it to cover an overrunning record, and that the final segment's duration should carry the last group's content end.Not changed: the timeline wire format is replaced outright with no versioning or migration, which the review also flagged. Nothing consumes the timeline yet, so there is nothing to stay compatible with.
(Written by Opus 5)