drawElement self-verify PSNR gate cannot detect missing low-contrast content

[miguel-heygen]

Describe the bug

The drawElement self-verify gate cannot detect a missing low-contrast element, because
PSNR is magnitude-weighted. An entire element can be absent from a rendered frame and
still score well above the pass threshold, so the damaged frame ships with a clean exit
code.

The gate is in packages/producer/src/services/render/stages/captureStreamingStage.ts:341:

db = await psnrDb(buf, truth);
...
if (db < verifyMinDb) { throw new DrawElementVerificationError(...) }

psnrDb takes FFmpeg's average: PSNR across components
(packages/engine/src/utils/psnr.ts:13), and verifyMinDb defaults to 32
(psnr.ts:43). PSNR falls as the magnitude of the difference grows. A whole element
that is merely low-contrast against its background produces a small magnitude over a
modest area, so it scores high and passes.

The failure mode this permits is the worst kind we can ship: a video that is wrong, with
no warning and exit code 0. High-contrast damage in the same frame is caught, which is
why this reads as "some elements render and some silently do not" rather than as a
general breakage.

Steps to reproduce

No project needed. This measures the gate itself against synthetic 4K frames, using the
same comparison psnrDb performs.

#!/usr/bin/env bash
set -uo pipefail
FF="${FF:-ffmpeg}"; W=/tmp/psnr; rm -rf $W; mkdir -p $W; cd $W
BG="0x12122E"; BAND=324   # 3840x324 = 1,244,160 px = 15.0% of a 4K frame

mkframe(){
  if [ -z "$2" ]; then
    $FF -v error -y -f lavfi -i "color=c=$BG:s=3840x2160:d=1" -frames:v 1 "$1"
  else
    $FF -v error -y -f lavfi -i "color=c=$BG:s=3840x2160:d=1" \
      -vf "drawbox=x=0:y=900:w=3840:h=$BAND:color=$2:t=fill" -frames:v 1 "$1"
  fi
}
cmp_psnr(){ $FF -hide_banner -i "$1" -i "$2" -lavfi psnr -f null - 2>&1 \
            | grep -o 'average:[^ ]*' | head -1; }

mkframe truth.jpg ""
for spec in "0x161632:~4/255 luma delta" "0x1E1E42:~10/255 luma delta" \
            "0x262556:~22/255 luma delta" "0xFFFFFF:white (control)"; do
  col="${spec%%:*}"; label="${spec#*:}"
  mkframe damaged.jpg "$col"
  db=$(cmp_psnr damaged.jpg truth.jpg)
  printf "  %-26s %-20s %s\n" "$label" "$db" \
    "$(awk -v d="${db#average:}" 'BEGIN{print (d+0>=32)?"PASSES 32 dB gate -> damage ships":"fails gate -> caught"}')"
done

truth.jpg is the background with the panel missing; damaged.jpg is the correct frame.
The PSNR between them is exactly what the gate computes when the panel fails to draw.

Expected behavior

A frame missing an entire visible element fails verification, regardless of that
element's contrast against its background.

Actual behavior

  ~4/255 luma delta          average:44.141763    PASSES 32 dB gate -> damage ships
  ~10/255 luma delta         average:35.739625    PASSES 32 dB gate -> damage ships
  ~22/255 luma delta         average:31.063362    fails gate -> caught
  white (control)            average:10.743118    fails gate -> caught

An element covering 15% of a 4K frame can go completely missing and still score 44 dB,
12 dB clear of the floor. The gate only starts catching it somewhere around a 12/255
luma delta at that coverage, and the threshold moves with area: a smaller element needs
proportionally more contrast before the gate notices it is gone.

Two separate holes

1. The metric is wrong for the question. PSNR answers "how far off are these pixels
on average", but the gate is asking "did any content go missing". Those come apart
precisely when the missing content is low-contrast. Raising HF_DE_VERIFY_MIN_DB is not
a fix: 44 dB would need a floor above 44, which would then reject ordinary JPEG encoder
noise. A presence-sensitive check is needed instead, for example a per-region or
worst-block comparison rather than a whole-frame average, so a fully wrong region cannot
be averaged away by a correct background.

2. The gate does not run on most frames. It is guarded by
const truth = session.deVerifyFrames?.get(idx) (captureStreamingStage.ts:337), so
frames without a sampled truth image are not checked at all. Even damage severe enough
to fail the threshold ships if it lands on an unsampled frame.

Scope of what is proven here

This demonstrates that the verification gate is blind to this class of damage. It does
not explain why drawElement omits a low-contrast element in the first place. Those are
separate bugs, and this one is worth fixing on its own: without a gate that can see the
damage, the underlying drop stays invisible in production no matter what causes it.

Until the metric is fixed, it is worth considering whether the streaming path should
fail loudly rather than proceed when it cannot verify a frame, since a hard failure is
recoverable and a silently wrong deliverable is not.

Environment

FFmpeg   7.0.2-static (linux x64)
Node.js  v22.22.2
Gate     packages/engine/src/utils/psnr.ts        resolveDeVerifyMinDb() default 32
Call     packages/producer/src/services/render/stages/captureStreamingStage.ts:341
Also     packages/engine/src/services/parallelCoordinator.ts:613 (same gate, disk path)

[miguel-heygen]

There is already a precedent in the repo for the policy this issue is asking for, and it
is worth copying rather than re-deciding.

packages/producer/src/services/render/videoFrameCoverage.ts exists for exactly this
failure shape and resolves it by aborting the render:

Video "<id>" captured 41 of expected 69 frames (coverage 59.4%, threshold 95.0%).
check/snapshot may pass while the encoded MP4 renders this clip blank — aborting render
to prevent shipping a wrong MP4.

The reasoning encoded there — that a green check plus a wrong MP4 is worse than a
failed render — is the same reasoning that applies here. So the question is not whether
failing closed is acceptable, it is why this path does not do it when the video path
already does.

Two things to take from that file:

1. It fails closed by default (threshold 0.95) with an explicit opt-out, rather than
   warning and continuing.
2. Its own header comment already documents a sibling gap it cannot see: the
   syncTimedElementVisibility case, where "147 clips fails, 3 succeeds" and the
   coverage gate "cannot directly observe" the damage. That is left as follow-up work.
   Between that note, this issue, and the other open one, there are now three separate
   known ways to ship a wrong MP4 with a clean exit code, which suggests the missing
   piece is a shared fail-closed policy rather than three more one-off gates.

One caution on copying it verbatim: its error advertises HF_VIDEO_COVERAGE_THRESHOLD=0
as the escape hatch, which tells a blocked user that the documented way forward is to
disable the check and ship the wrong output. Any new gate should point at the cause
instead of at its own off switch.