Skip to content

feat(realtime): lucy-2.5 + optional sampleFrameData on set/setImage#135

Draft
tomershlasky wants to merge 10 commits into
mainfrom
feat/sdk-reference-frame
Draft

feat(realtime): lucy-2.5 + optional sampleFrameData on set/setImage#135
tomershlasky wants to merge 10 commits into
mainfrom
feat/sdk-reference-frame

Conversation

@tomershlasky
Copy link
Copy Markdown
Contributor

@tomershlasky tomershlasky commented May 13, 2026

Summary

  • Adds the lucy-2.5 realtime model (shared/model.ts, exposed via models.realtime("lucy-2.5") and the registry).
  • Adds optional sampleFrameData (Blob | File | base64 string | null) to realtime.set() and realtime.setImage(), serialized as a new sample_frame_data field on the existing set_image WS message. Field is only emitted when the caller passes it — no behavior change for existing callers.
  • Adds a "capture & attach camera reference frame" toggle plus the lucy-2.5 option in the playground (packages/sdk/index.html).

Merge with main / LiveKit re-home

Main shipped the LiveKit migration (#122/#138) and observability orchestrator (#132) since this branch last picked up main. The legacy webrtc-connection.ts / webrtc-manager.ts modules are gone, so the original wiring was re-homed onto the new transport:

  • signaling-channel.ts#setImage now writes sample_frame_data on the outgoing set_image WS frame when provided.
  • stream-session.ts#setImage forwards via the existing ImageSetOptions plumbing.
  • realtime/client.ts#setImage accepts the public sampleFrameData (Blob/File/string/null) and converts to base64 at the boundary.
  • realtime/methods.ts#set does the same conversion at the helper boundary.
  • Public type lives on ImageSetOptions in realtime/types.ts; wire field on SetImageMessage.

Test cleanup: dropped the old WebRTCConnection/WebRTCManager-specific tests that no longer apply; added focused sampleFrameData coverage on mockSession.setImage in tests/realtime.unit.test.ts.

Test plan

  • pnpm -F @decartai/sdk build clean
  • pnpm -F @decartai/sdk lint and format:check clean
  • pnpm -F @decartai/sdk test — 210/210 pass (5 test files), incl. new sampleFrameData wire coverage
  • Live playground smoke test against /v1/stream: connect with lucy-2.5, toggle reference-frame checkbox, confirm sample_frame_data appears on the outgoing set_image WS frame in DevTools (and is absent when the checkbox is off)

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 13, 2026

Open in StackBlitz

npm i https://pkg.pr.new/@decartai/sdk@135

commit: 0b62195

@tomershlasky tomershlasky changed the title feat(realtime): add optional referenceFrame to set/setImage feat(realtime): lucy-2.5 + optional sampleFrameData on set/setImage May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant