Skip to content

Video extraction failures are swallowed, surfacing as a misleading coverage abort #3372

Description

@miguel-heygen

Describe the bug

A render aborts with a coverage error that names the wrong subsystem, because the real
failure — a per-source video extraction that produced zero frames — is swallowed by default
and never surfaces.

The reported symptom:

Video "bg-blur" captured 0 of expected 36 frames (coverage 0.0%, threshold 95.0%).
check/snapshot may pass while the encoded MP4 renders this clip blank — aborting render
to prevent shipping a wrong MP4. Set HF_VIDEO_COVERAGE_THRESHOLD=0 to disable this gate.

The coverage gate is not at fault. It is the only component still talking by the time the
clip arrives with no frames.

Steps to reproduce

Cold cache, one local file, one clip, default flags. The only unusual thing is a source
whose audio outlasts its video — which is what screen recorders, trimmed exports, and
image-plus-music renders routinely produce.

npx --yes hyperframes@0.8.4 init proj --example blank --non-interactive
cd proj && mkdir -p assets

# video stream 3s, audio 8s -> container reports duration 8s
ffmpeg -y -f lavfi -i "testsrc=size=640x360:rate=30:duration=3" \
       -f lavfi -i "sine=frequency=440:duration=8" \
       -c:v libx264 -pix_fmt yuv420p -c:a aac assets/tail.mp4

Set the root to data-duration="1.2" and use one clip whose media-start lands in the
audio-only tail:

<div class="clip" data-start="0" data-duration="1.2" data-track-index="0">
  <video id="bg-blur" src="assets/tail.mp4" data-media-start="5"></video>
</div>
npx --yes hyperframes@0.8.4 render --output out.mp4

Expected behavior

The render fails naming the actual cause: extraction produced no frames because the
requested media-start is past the end of the video stream.

Actual behavior

ffprobe container duration:     8.000000
ffprobe video stream duration:  3.000000

totalFramesExtracted":0
minVideoFrameCoverageRatio":0
captured 0 of expected 36 frames (coverage 0.0%, threshold 95.0%)

Reproduces every time on a fresh cache. Note the message is byte-identical to the field
report, including the frame count.

Root cause

resolveVideoExtractionPolicy() in
packages/producer/src/services/render/stages/extractVideosStage.ts:138 defaults
failureMode to "off":

const rawMode = env.HF_VIDEO_EXTRACTION_FAILURE_MODE?.trim().toLowerCase();
const failureMode: VideoExtractionFailureMode =
  rawMode === "observe" || rawMode === "enforce" ? rawMode : "off";

So every per-source extraction failure is recorded and then dropped. ffmpeg -ss 5 into a
video-less tail exits 0 having written nothing, which raises zero_output — and with
failureMode: "off" that error goes nowhere, not even to stderr. The clip reaches the
coverage gate with no ExtractedFrames entry, scores 0/expected, and the user is handed
a message about capture coverage that mentions nothing about extraction.

The escape hatch the message advertises, HF_VIDEO_COVERAGE_THRESHOLD=0, would tell a
blocked user to disable a correctness gate to work around a bug that gate did not cause.

Two secondary defects that funnel into the same message

1. A cache entry with zero frames is a valid hit.
rehydrateCacheEntry (packages/engine/src/services/extractionCache.ts:481) trusts the
.hf-complete sentinel and takes whatever readdirSync returns, with no check against
totalFrames:

const files = readdirSync(entry.dir)
  .filter((f) => f.startsWith(FRAME_FILENAME_PREFIX) && f.endsWith(suffix))
  .sort();

An entry whose frames were removed by any per-file cleanup keeps its sentinel and rehydrates
as a hit with zero frames, so every subsequent render of that project aborts identically.
That is a plausible explanation for field reports of "happens every time" with no user-
discoverable fix — the poison is on disk, not in the composition.

2. The extraction window is derived from container duration.
The container above reports 8s while the video stream is 3s. A media-start computed from the
duration HyperFrames itself reports can therefore land somewhere that cannot deliver frames.

Suggested fix

Stop swallowing per-source extraction failures. A video with a recorded extraction error
should fail with that error — zero_output, ffmpeg_failed, source_missing — before the
coverage gate is reached. Whether that means flipping the default from "off" to "enforce"
or surfacing the error separately is a judgement call, but a silently discarded error is what
makes this class of failure unreadable.

Then two guards:

  • Treat a sentineled cache entry whose frame count is zero (or below its recorded
    totalFrames) as a miss and re-extract.
  • Clamp media-start against playable video-stream duration, with an authoring error when it
    lands past the last frame.

Leave the coverage gate alone. It is behaving correctly and it is the only reason these
silent extraction failures are visible at all.

Environment

hyperframes  0.8.4
Node.js      v22.23.1 (darwin arm64)
Affected     packages/producer/src/services/render/stages/extractVideosStage.ts:138
             packages/engine/src/services/extractionCache.ts:481

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions