Skip to content

reformat: fix u32 row offset wrap in slow path#3202

Open
rootvector2 wants to merge 1 commit into
AOMediaCodec:mainfrom
rootvector2:fix-reformat-row-offset-overflow
Open

reformat: fix u32 row offset wrap in slow path#3202
rootvector2 wants to merge 1 commit into
AOMediaCodec:mainfrom
rootvector2:fix-reformat-row-offset-overflow

Conversation

@rootvector2
Copy link
Copy Markdown
Contributor

Summary

In the slow YUV-to-RGB fallback avifImageYUVAnyToRGBAnySlow() (src/reformat.c:686-692), the per-row Y/U/V/A pointers are computed with j * yRowBytes style multiplications where both operands are uint32_t. When the plane size exceeds 4 GiB (e.g. a 70000x70000 8-bit YUV400 image, or any combination whose height * yRowBytes exceeds UINT32_MAX), the product wraps around in 32 bits before being promoted to size_t for the pointer arithmetic. The loop then reads from a different row inside the same plane buffer, producing silently corrupted output.

The neighbouring RGB destination pointers in the same loop body already use (size_t)j * rgb->rowBytes, and the same pattern was applied across the file in c79a400 and f17110d. This change applies the same cast to the four YUV/A reads that were missed.

-        const uint8_t * ptrY8 = &yPlane[j * yRowBytes];
-        const uint8_t * ptrU8 = uPlane ? &uPlane[(uvJ * uRowBytes)] : NULL;
-        const uint8_t * ptrV8 = vPlane ? &vPlane[(uvJ * vRowBytes)] : NULL;
-        const uint8_t * ptrA8 = aPlane ? &aPlane[j * aRowBytes] : NULL;
+        const uint8_t * ptrY8 = &yPlane[(size_t)j * yRowBytes];
+        const uint8_t * ptrU8 = uPlane ? &uPlane[((size_t)uvJ * uRowBytes)] : NULL;
+        const uint8_t * ptrV8 = vPlane ? &vPlane[((size_t)uvJ * vRowBytes)] : NULL;
+        const uint8_t * ptrA8 = aPlane ? &aPlane[(size_t)j * aRowBytes] : NULL;

Reproduction

Built a small driver that constructs an avifImage with width=2, height=4, AVIF_PIXEL_FORMAT_YUV420, points the Y plane at an mmap region (virtual, only a few pages touched physically) and sets yuvRowBytes[Y] = 0x60000000. Distinct sentinel bytes are written at offsets 0, 0x60000000, 0xC0000000, the true row-3 offset 0x120000000 (0xCC), and the wrapped uint32_t row-3 offset 0x20000000 (0xBB). avifImageYUVToRGB is then called with AVIF_CHROMA_UPSAMPLING_BILINEAR and alphaPremultiplied=AVIF_TRUE to force the slow path.

Pre-patch output:

row0_r=a0 row1_r=a1 row2_r=a2 row3_r=bb

i.e. row 3 of the output mirrors the wrapped-offset sentinel, not the true row-3 sentinel.

Post-patch output:

row0_r=a0 row1_r=a1 row2_r=a2 row3_r=cc

i.e. row 3 reads from the correct offset.

Test plan

  • Library builds without warnings.
  • Confirmed with the mmap-backed driver above: pre-patch reads 0xBB (wrapped offset) for the last row, post-patch reads 0xCC (true offset).
  • Existing avifyuv_limited / avifyuv_rgb tests still pass (aviftest is unrelated and fails only because the local build is configured without an AV1 codec).
  • No regression in the normal (sub-4 GiB) case: smaller-image YUV-to-RGB conversions through the slow path still produce identical output.

In the slow YUV-to-RGB fallback in src/reformat.c, the Y/U/V/A row
pointers were computed with 'j * yRowBytes' style multiplications
where both operands are uint32_t. For images whose plane size exceeds
4 GiB the product wraps around and the loop reads from a different
row of the same buffer, producing silently corrupted output.

Cast the row index to size_t before the multiplication so the offset
is computed in size_t arithmetic, matching what is already done for
the RGB destination pointers a few lines below and the same pattern
applied in c79a400.
@rootvector2 rootvector2 force-pushed the fix-reformat-row-offset-overflow branch from d14b742 to 87cc17a Compare May 18, 2026 16:51
@rootvector2 rootvector2 changed the title Fix uint32_t overflow of row offsets in avifImageYUVAnyToRGBAnySlow reformat: fix u32 row offset wrap in slow path May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant