feat(agent-core-v2): add visual model assignment for image inspection - #2754
feat(agent-core-v2): add visual model assignment for image inspection#2754zikzak-ai wants to merge 1 commit into
Conversation
Mirror the experimental `[secondary_model]` slot as a new `[visual_model]`
configuration section so users can pin a vision-capable model for image /
screenshot / video inspection tasks even when their main coding model is
text-only.
- `[visual_model]` config section + `KIMI_VISUAL_MODEL` /
`KIMI_VISUAL_EFFORT` env overrides, registered alongside the other
kosong config sections (parallel to `SECONDARY_MODEL_SECTION`).
- `visualModelOverlay` synthesizes the derived `__visual__` registry
entry when the recipe carries patch fields (mirror of
`secondaryModelOverlay`); the derived entry lives only in memory and
is stripped from `config.toml` writes.
- `visual-model` experimental flag (`KIMI_CODE_EXPERIMENTAL_VISUAL_MODEL`)
gates the resolver; off by default.
- `resolveVisualModel` / `resolveVisualBinding` /
`buildVisualModelDescriptions` / `visualDisplayModel` /
`stripVisualModelParameter` / `wrapVisualModelError` mirror the
subagent resolver family.
- `AgentMediaToolsRegistrar` now consults `resolveVisualModel`: when the
caller's model is text-only but a vision-capable visual model is
configured, the media tools register against the visual model's
capabilities and requester so `ReadMediaFile` stays available. When
unset, behavior is unchanged.
- Regenerates `docs/config-manifest.toml` (now lists `visualModel` and
the `visualModelOverlay`).
- Bilingual VitePress docs (`docs/{en,zh}/configuration/config-files.md`)
describe the new section, env vars, experiment flag, and the
unchanged-when-unset contract.
- Vitest coverage: `visualModelOverlay.test.ts` (apply / strip / no
collision with `__secondary__`) and `session/visual/configSection.test.ts`
(resolution + unset-fallback + display model + schema strip + error
wrapping). The existing `read-media.test.ts` registrar construction
site is updated for the two new constructor deps.
Closes MoonshotAI#2750
🦋 Changeset detectedLatest commit: 605bb69 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 605bb69005
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const useVisual = !callerHasMedia && visualHasMedia; | ||
| const capabilities = useVisual ? visualCapabilities! : callerCapabilities; |
There was a problem hiding this comment.
When the experiment is enabled and the caller model has no media capability but [visual_model] points to a vision model, this branch registers ReadMediaFile using the visual model's capabilities. The tool execution still writes image/video ContentParts into the normal tool-result history, and llmRequesterService.resolveRequest sends the following request with the caller profile's requester, so the advertised text-only-main-model scenario sends media payloads to the text-only model instead of to the visual model. Keep the tool gated on the caller model until the media read is summarized through the visual model, or route the follow-up request to that visual binding.
Useful? React with 👍 / 👎.
|
|
||
| import { VISUAL_MODEL_FLAG_ID } from './flag'; | ||
|
|
||
| export type VisualModelChoice = AgentModelPreference; |
There was a problem hiding this comment.
VisualModelChoice aliases AgentModelPreference, whose values are primary | secondary, but this resolver and the exported schema/documentation use visual. A TypeScript caller cannot pass the intended "visual" value to resolveVisualBinding without a cast, while "secondary" is accepted and is treated as the visual path because the branch only checks for "primary". Define this type from VISUAL_MODEL_CHOICE_SCHEMA or a primary | visual literal union so the public contract matches runtime behavior.
Useful? React with 👍 / 👎.
1 participant
Related Issue
Closes #2750
Problem
Many coding models are text-only — they cannot inspect images. Web and mobile development regularly requires visual inspection (screenshots, UI states, layout renders, image diffs of a rendered page). Today there is no way for a user to pin a specific, vision-capable model for these visual tasks; they fall back to automatic / current-model selection, which is often a text-only model. See the linked issue for the full context.
What changed
Mirrors the existing experimental
[secondary_model]slot as a new[visual_model]configuration section, so a user can pin a vision-capable companion model for image / screenshot / video inspection. The design copies the secondary-model shape verbatim — config section + env overrides + experiment flag + derived-entry overlay + resolver family — and wires the resolver into the media-tools registrar.Config + overlay + flag
[visual_model]config section registered inpackages/agent-core-v2/src/app/kosongConfig/configSection.ts(parallel toSECONDARY_MODEL_SECTION). Acceptsmodel,default_effort, and every[models."<alias>".overrides]field as a patch.KIMI_VISUAL_MODEL/KIMI_VISUAL_EFFORTenv overrides (priority overconfig.toml).packages/agent-core-v2/src/app/kosongConfig/visualModelOverlay.tssynthesizes the derived__visual__registry entry when the recipe carries patch fields (base copy, patch merged intooverrideswith patch winning conflicts,aliasesdropped). The derived entry lives only in memory and is stripped fromconfig.tomlwrites — including adefault_modelpointer at the derived id. Mirror ofsecondaryModelOverlay.packages/agent-core-v2/src/session/visual/flag.tsregisters thevisual-modelexperimental flag (KIMI_CODE_EXPERIMENTAL_VISUAL_MODEL), default off, surfacecore.Resolver family
packages/agent-core-v2/src/session/visual/configSection.tsexports the visual-model resolver family, mirroringsession/subagent/configSection.ts:resolveVisualModel(config, flags)— returns the configured recipe when the experiment is on,undefinedotherwise.resolveVisualBinding(config, flags, own, requested?)— resolves which model a visual task binds to. Unset = caller's model (no behavior change). Explicit'primary'forces the caller's model even when a visual model is configured.buildVisualModelDescriptions(...)— the "Available models for visual inspection" block, with capability suffixes, mirroringbuildSubagentModelDescriptions.visualDisplayModel(config, boundAlias)— resolves the derived__visual__id back to the recipe's base alias for display, flag-independent so an already-persisted derived binding still resolves after the experiment is switched off.stripVisualModelParameter(...)— strips the no-opmodelparameter from a visual-task tool's advertised schema while the experiment is off.wrapVisualModelError(...)— wraps a missing-alias failure with a hint pointing at[visual_model].model/KIMI_VISUAL_MODEL.Wiring
packages/agent-core-v2/src/agent/media/mediaToolsRegistrar.tsnow injectsIConfigService+IFlagServiceand consultsresolveVisualModelon every refresh. When the caller's model is text-only (!image_in && !video_in) but a vision-capable visual model is configured and the experiment is on, the registrar registersReadMediaFileagainst the visual model's capabilities and video uploader so the LLM keeps the tool available for visual inspection. When the visual model is unset, the registrar's behavior is identical to before (caller's capabilities gate, caller's requester binds). The state key now includes the visual model alias + capability signature so a visual-model change re-runs registration.The actual end-to-end routing of image content to the visual model (e.g. via a subagent spawn for the inspection) is a follow-up enhancement that this PR enables — the configuration slot, resolver, and registrar gate are the foundation.
Index exports
packages/agent-core-v2/src/index.tsre-exports the new symbols (VISUAL_MODEL_SECTION,VISUAL_MODEL_ENV,VISUAL_MODEL_EFFORT_ENV,VisualModelConfigSchema,visualModelEnvBindings,VISUAL_DERIVED_MODEL_ID,visualModelOverlay,visualModelPatch,VISUAL_MODEL_FLAG_ID,VISUAL_MODEL_FLAG_ENV,visualModelFlag,resolveVisualModel,resolveVisualBinding,visualDisplayModel,buildVisualModelDescriptions,stripVisualModelParameter,wrapVisualModelError,VisualModelChoice) and side-effect-imports the overlay + flag + resolver so the registrations run on first import.Docs + manifest
## visual_modelsection indocs/en/configuration/config-files.mdanddocs/zh/configuration/config-files.md, mirroring the## secondary_modelsection (table, patch semantics, env overrides, experiment flag, advisory validation note).packages/agent-core-v2/docs/config-manifest.tomlregenerated viapnpm --filter @moonshot-ai/agent-core-v2 gen:config-manifest— now listsvisualModeland thevisualModelOverlay.Tests
packages/agent-core-v2/test/app/kosongConfig/visualModelOverlay.test.ts(9 tests, mirror ofsecondaryModelOverlay.test.ts): apply path (no-op, pointer-only, patch synthesis, missing pointer, never-derive-from-derived, no collision with__secondary__) and strip path (removes derived entry, leaves other domains, rolls backdefaultModelpointer).packages/agent-core-v2/test/session/visual/configSection.test.ts(19 tests):resolveVisualModel(disabled flag, unset, set),resolveVisualBinding(unset fallback, disabled-flag fallback, pointer-only binding, derived-entry binding + display, explicitprimaryoverride),visualDisplayModel(passthrough, derived-id resolution, fallback after recipe removal),stripVisualModelParameter(no-op, removesmodel+required, no mutation),wrapVisualModelError(caller-own passthrough, non-CONFIG_INVALID passthrough, mismatched-details passthrough, missing-alias wrap with[visual_model]hint, derived-id wrap).packages/agent-core-v2/test/agent/media/tools/read-media.test.tsupdated for the two newAgentMediaToolsRegistrarconstructor deps (StubConfigService+stubFlag(false)).Changeset
.changeset/visual-model-assignment.md—@moonshot-ai/kimi-code: minor(user-visible config + behavior change in the CLI).Checklist
gen-changesetsskill (changeset added for@moonshot-ai/kimi-code).gen-docsskill (bilingual VitePress docs updated; manifest regenerated).pnpm lint— 0 errors (only pre-existing warnings in unrelatedpackages/minidb).pnpm --filter @moonshot-ai/agent-core-v2 typecheck— clean.pnpm --filter @moonshot-ai/agent-core-v2 exec vitest runfor the touched tests — all pass (visual overlay + visual resolver + read-media + secondary-model + secondary-model-warning).