Skip to content

Artifact validation only checks the file is non-empty, so a truncated render ships as success #3395

Description

@miguel-heygen

Describe the bug

A multi-worker render can deliver a truncated video while reporting success, because
"artifact validation" only checks that the output file exists and is non-empty.

Reported with a full packet:

EXPECTED / ACTUAL: 1566 frames at 52.2s
                 / command reported 1566 captured and Render complete,
                   but artifact contained 1240 frames at 41.333s
EXACT ERROR: none; artifact validation incorrectly passed after multi-worker encode
OUTCOME: output truncated; fallback succeeded with --workers 1
WORKAROUND: --workers 1
COMPOSITION_STRUCTURE: elements: video=2 audio=1 ... attributes: mix-blend-mode,
  data-duration, data-start

326 frames — about 11 seconds — are missing from the delivered file. The CLI printed
Render complete.

The validation gap

ArtifactTransaction.validate()
(packages/producer/src/services/render/artifactTransaction.ts:93):

validate(): void {
  if (this.kind === "file") {
    assertReadableNonEmptyFile(this.stagingPath);
    return;
  }
  ...
}

For a video that is the whole check: readable, and larger than zero bytes. Duration is
never compared against the composition, and frame count is never compared against the
frames the pipeline just reported capturing.

renderOrchestrator.ts:3796 calls it and then emits the checkpoint
"artifact validated", which reads — to a maintainer and to anyone parsing the trace — as
though the artifact was checked against what was asked for.

Why this one is worth fixing

The information needed to catch it was already in hand at the moment it passed. The same run
reported 1566 frames captured; the file contained 1240. A comparison of two numbers
the pipeline already holds would have turned a silently wrong deliverable into a failed
render.

This is also the failure mode most likely to reach an end viewer. A wrong pixel is subtle; a
video that stops 11 seconds early is not, and by then it has been published.

Suggested fix

At validate(), for file artifacts of a video format, additionally assert:

  1. container duration is within a frame or two of totalFrames / fps, and
  2. the decoded frame count matches the frames the capture stage reported.

Both are one ffprobe call on a file the pipeline has just written, and the expected values
are already carried in the job. Failing here is safe: the artifact is still in its staging
directory and has not been committed to the destination, so a hard failure loses nothing a
retry cannot redo.

Worth considering whether the checkpoint should be renamed until it does this — artifact validated currently overstates what happened.

Scope note

I have not reproduced the truncation itself, which appears to need multi-worker encode on a
particular composition; the reporter's own workaround was --workers 1. The validation
gap above is independent of that
and is verifiable by reading: whatever causes the
truncation, nothing downstream is positioned to notice it.

The underlying multi-worker truncation deserves its own investigation, and this issue is
deliberately about the gate that let it through.

Environment

hyperframes  0.8.6
Affected     packages/producer/src/services/render/artifactTransaction.ts:93
             packages/producer/src/services/renderOrchestrator.ts:3796,3804

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions