Skip to content

fix(multimodal): pin the Qwen2-VL XLA loader contract and document all three outcomes - #987

Merged
inureyes merged 1 commit into
mainfrom
fix/issue-966-qwen2vl-xla-loader-contract
Jul 31, 2026
Merged

fix(multimodal): pin the Qwen2-VL XLA loader contract and document all three outcomes#987
inureyes merged 1 commit into
mainfrom
fix/issue-966-qwen2vl-xla-loader-contract

Conversation

@inureyes

Copy link
Copy Markdown
Member

What was broken

multimodal::host_preprocessor::tests::xla_loader_keeps_text_and_unqualified_vlm_image_capability_false has failed deterministically on main since edc0ebbb6 (#915). It loops over ["llama", "qwen2_vl"] asserting load_xla_image_preprocessor(...) returns Ok(None), but #915 made qwen2_vl return Err(InvalidConfig("Qwen2-VL XLA image execution requires the xla-iree feature")) when the feature is absent, so the .unwrap() panics. That commit edited the same test file, adding an import and two export tests, and left this loop untouched, so the contract and the test that pins it diverged inside a single change to a single file.

This is the third deterministically-failing test found on main in one week, after the two repaired in #946 and #953. It is the first one the nightly verification job added in #953 should catch on its own.

Which side is right

The production behavior stays; the test changes. The reasoning, recorded here so it is not relitigated:

Qwen2-VL has no MLX vision fallback. The arm immediately above already rejects MLXCEL_XLA_VISION_BACKEND=host for it with exactly that reason, so in a build without xla-iree there is no code path that can embed its images. Ok(None) for such a checkpoint is indistinguishable from a text-only checkpoint, so a session would start with every image silently ignored, which is a worse outcome than a named failure at load.

Failing at load also takes away nothing that worked. xla-iree = ["xla-backend", "mlxcel-xla/iree"], and without mlxcel-xla/iree the session's prefill and decode_step return mlxcel_xla::NOT_WIRED, whose text is "the OpenXLA inference session was built without the iree feature; rebuild ...". An xla-backend-only build cannot generate from any checkpoint at all, so there is no working configuration being broken here.

Feature absence already errors elsewhere in the same function: an explicit MLXCEL_XLA_VISION_BACKEND=iree returns Err for the same reason.

The counter-argument, addressed rather than dropped

Without xla-iree, LLaVA does not error. Unless the policy is explicitly iree it falls through to the host preprocessor and returns Ok(Some(host)). Qwen2-VL is the only family where a missing compile-time feature is fatal, and that asymmetry deserved an answer rather than silence.

It is deliberate and now documented where a reader will find it: LLaVA has a complete MLX host vision tower and projector to fall back to, Qwen2-VL has none. The doc comment on load_xla_image_preprocessor previously described only two of the three outcomes the function implements, which is precisely how the contract drifted from its test. It now covers all three (Ok(None), Ok(Some), Err), states which families reach each and why, and records the NOT_WIRED reasoning.

Changes

The capability-false loop drops qwen2_vl and gains mllama, so it still covers an unqualified VLM family rather than degrading to a text-only-only test, with an inline comment explaining why qwen2_vl is deliberately absent and where its contract is pinned instead.

A new xla_loader_rejects_qwen2_vl_without_the_iree_feature, gated #[cfg(not(feature = "xla-iree"))] because with the feature the same config takes the IREE load path, asserts the error variant and that its message both names the missing feature and carries the --features xla-iree rebuild instruction. Both callers surface this string verbatim into a startup failure, so the instruction is the only actionable part an operator receives; the error text was extended to carry it.

Validation

cargo test --release --lib --features metal,accelerate multimodal::host_preprocessor goes from 11 passed; 1 failed to 13 passed; 0 failed. cargo fmt --all -- --check and the cross-repo-ref guard are clean.

No production behavior changes: the only non-test edits are the doc comment and the error string.

Closes #966

…l three outcomes

`xla_loader_keeps_text_and_unqualified_vlm_image_capability_false` asserted `Ok(None)` for `qwen2_vl`, but #915 made that family return `Err` when the `xla-iree` feature is absent. The commit that changed the contract edited the same test file without touching this loop, so the test has failed deterministically on `main` ever since.

The production behavior is correct and stays. Qwen2-VL has no MLX vision fallback, which is why `MLXCEL_XLA_VISION_BACKEND=host` is already rejected for it, so without `xla-iree` no path can embed its images. `Ok(None)` there is indistinguishable from a text-only checkpoint and would start a session that silently ignores every image. Failing at load takes nothing away that worked: `xla-iree` also enables `mlxcel-xla/iree`, and without it `prefill` and `decode_step` return `NOT_WIRED`, so such a build cannot generate from any checkpoint.

`qwen2_vl` leaves the capability-false loop and is replaced by `mllama`, so the loop still covers an unqualified VLM family rather than only a text-only one, and a new `#[cfg(not(feature = "xla-iree"))]` test pins the error. That test asserts the message both names the missing feature and carries the rebuild instruction, because both callers surface the string verbatim and the instruction is the only actionable part an operator gets.

The doc comment on `load_xla_image_preprocessor` described only two of the three outcomes it implements, which is how the contract and its test diverged in the first place. It now covers all three and records why the Qwen2-VL and LLaVA asymmetry is deliberate: LLaVA has a complete MLX host tower to fall back to, Qwen2-VL does not.

Closes #966
@inureyes
inureyes force-pushed the fix/issue-966-qwen2vl-xla-loader-contract branch from a1bcbfa to 044a804 Compare July 31, 2026 02:42
@inureyes inureyes added the status:done Completed label Jul 31, 2026
@inureyes
inureyes merged commit ac793d6 into main Jul 31, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:done Completed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: Reconcile the Qwen2-VL XLA loader contract with its failing capability test

1 participant