New RawImage control: full-bandwidth pixel-frame streaming over DataChannels#6674
Merged
Conversation
…hannels Generalize the matplotlib raw-frame pipeline into a core control for Pillow output, numpy arrays, camera frames and procedural graphics. Updating Image.src resends the whole blob through the msgpack protocol and re-decodes it on every change; RawImage streams frames over a dedicated DataChannel with awaitable render() methods that resolve on the client's frame-applied ack, so producer loops self-pace to display speed. - Same wire format as MatplotlibChartCanvas: 0x04 raw premultiplied RGBA8888 on local transports, 0x01 encoded (auto PNG fallback, encoded off-loop) on remote web, 0x03 clear, 0xFF ack - render() accepts PIL images (duck-typed) and __array_interface__ arrays; Pillow and numpy stay optional via lazy imports - Premultiplication runs in Pillow's C loops (ImageChops.multiply per band) with a getextrema() opaque fast-path; numpy path reuses the Agg backend's uint16 math - Last frame is retained and replayed on widget remount so the image survives page rebuilds like Image.src - Dart side holds at most one live ui.Image (web-safe) and renders through Flutter's RawImage for Image-consistent fit/filter_quality/ scale semantics - Unit tests for packet layout and premultiply math (PIL vs numpy parity), 8 integration tests with goldens incl. a premultiply proof - Examples: plasma (fps counter, detail slider), paint (supersampled anti-aliasing, dirty-flag render loop), mandelbrot (click-to-zoom, letterbox-aware tap mapping), game_of_life (pointer cell drawing); registered in the gallery catalog - Docs page with RawImage vs Image guidance
Cover displaying regular encoded images (not just synthetic frames): - photo_viewer example: downloads network JPEGs and displays the bytes with render_encoded, with prev/next navigation and a byte cache - Integration tests rendering real PNG (with transparency) and JPEG files from the test assets straight from disk, with goldens - Docs: photo viewer section on the RawImage page
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Deploying flet-website-v2 with
|
| Latest commit: |
3095a7c
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://07106316.flet-website-v2.pages.dev |
| Branch Preview URL: | https://raw-image.flet-website-v2.pages.dev |
DataChannel has no API entry in the docs data, so the :class: role can't resolve; use inline code instead.
Add types pages and sidebar entries so :class: cross-references to flet.DataChannel resolve; restore the role in the RawImage docstring.
…let-web client send_data_channel_frame in the flet-web FastAPI transport dereferenced the send queue after the WebSocket send loop cleared it on disconnect; drop the frame instead, mirroring FletSocketServer. Dropped frames also mean the frame-applied ack never arrives, which would leave awaited render() loops hanging forever (Session.close does not cancel user tasks). Add RawImage.ack_timeout (default 10s): a render call now raises TimeoutError when the ack does not arrive, the same exception producer loops already catch to exit. The abandoned ack future is withdrawn so a late ack cannot resolve the wrong frame's wait. Also make the local-transport probe safe on detached controls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generalizes the matplotlib raw-frame pipeline (#6673) into a core
ft.RawImagecontrol — a persistent pixel surface the app pushes frames onto over a dedicated DataChannel, for Pillow output, numpy arrays, camera frames and procedural graphics at interactive frame rates.Updating
Image.srcresends the whole blob through the msgpack control-patch protocol and forces a full client-side re-decode on every change.RawImageinstead streams frames over a DataChannel with awaitablerender()methods that resolve on the client's frame-applied ack, so a plainwhile True: await ri.render(...)loop self-paces to display speed.Design
MatplotlibChartCanvas:0x04raw premultiplied RGBA8888[w u32 LE][h u32 LE][pixels],0x01encoded image bytes,0x03clear,0xFFack (Dart → Python).local_data_transportconnections (desktop,flet run, Pyodide); automatic PNG fallback encoded off the event loop on remote flet-web. Opaque frames PNG-encode losslessly; non-opaque premultiplied raw input falls back to raw with a documented caveat.render(pil_or_array, premultiplied=False)(PIL duck-typed,__array_interface__for numpy),render_rgba(w, h, bytes),render_encoded(png_jpeg_webp_bytes),clear();fit,filter_quality,scale,ready_timeoutproperties.ImageChops.multiplyper band) with agetextrema()opaque fast-path; the numpy path reuses the Agg backend's uint16 math. Pillow and numpy remain optional via lazy imports.Image.src.ui.Image(web-safe by construction) and renders through Flutter's built-inRawImage, giving Image-consistentfit/layout semantics.Tests
fit,clear, and an ack-backpressure burst.Examples & docs
Five gallery examples (registered in
gallery.yamlunder Displays/RawImage): photo_viewer (encoded network JPEGs), plasma (FPS counter, detail slider), paint (supersampled anti-aliasing, dirty-flag render loop), mandelbrot (click-to-zoom, letterbox-aware tap mapping), game_of_life (pointer cell drawing). Newrawimage.mddocs page with "RawImage vs Image" guidance and a generated docs image.Summary by Sourcery
Introduce a RawImage control that streams pixel frames over a dedicated data channel for high-frequency image updates, with full client/server plumbing, tests, examples, and docs.
New Features:
Enhancements:
Documentation:
Tests:
Chores: