Skip to content

fix(xla): bound diagnostic local-task threads - #945

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-932-diagnostic-local-task-threads
Jul 27, 2026
Merged

fix(xla): bound diagnostic local-task threads#945
inureyes merged 1 commit into
mainfrom
feature/issue-932-diagnostic-local-task-threads

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

  • add a diagnostics-only IREE flag helper that selects one local-task topology group
  • use the host-default pthread stack instead of the runtime exact-minimum request that fails on this host
  • cache the process-global parse result so repeated diagnostic setup is deterministic
  • compile and export the helper only under the diagnostics feature; production startup remains unchanged

Validation

  • cargo fmt --all -- --check
  • cargo test -p mlxcel-xla diagnostic_flags --lib: 2 passed
  • cargo check -p mlxcel-xla --features diagnostics: passed with pre-existing warnings only
  • cargo clippy -p mlxcel-xla --features diagnostics --no-deps: passed with pre-existing warnings only
  • git diff --check

Runtime boundary

The mlxcel-xla library test binary intentionally does not link IREE runtime archives; final native linkage and execution are validated by the family diagnostic executable in #919 immediately after this foundation lands.

Refs #932

@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:high High priority area:architecture Architecture and code structure changes area:core mlxcel-core: MLX FFI, primitives, KV cache, layers area:inference Generation, sampling, decoding (incl. speculative, DRY) status:done Completed and removed status:review Under review labels Jul 27, 2026
@inureyes
inureyes merged commit ec6fbb6 into main Jul 27, 2026
5 checks passed
@inureyes
inureyes deleted the feature/issue-932-diagnostic-local-task-threads branch July 27, 2026 02:22
inureyes added a commit that referenced this pull request Jul 27, 2026
The Molmo v1 reference gate created its production IREE preprocessor
without first applying the diagnostics-only local-task thread
configuration added in #945. `configure_diagnostic_local_task_threads`
had no callers on this branch, while the Gemma3 runner already applies
it, so the Molmo runner aborted before any comparison ran:

    xla_aux_create failed (status 13):
    iree/base/threading/thread_pthreads.c:159: INTERNAL;
    thread creation failed with 22

IREE parses its process-global flag registry when the first instance is
created, so the call has to precede `load_iree` to have any effect.

With this, the pinned 5.0 GB checkpoint gate completes end to end for the
first time on GB10: processor stages exact, patch embedding, selected
layer 14, projector, both prepared sparse-add merge stages, layer-0 KV,
all-layer KV and prefill logits all pass, leaving 4 of 1,181,696 values
outside tolerance at selected layer 21.

Refs #870
inureyes added a commit that referenced this pull request Jul 27, 2026
The Molmo2 vision parity gate created its IREE diagnostic projector
without first applying the diagnostics-only local-task thread
configuration added in #945. `configure_diagnostic_local_task_threads`
had no callers on this branch, so the gate aborted before any
comparison ran:

    xla_aux_create failed (status 13):
    iree/base/threading/thread_pthreads.c:159: INTERNAL;
    thread creation failed with 22

IREE parses its process-global flag registry when the first instance is
created, so the call has to precede the projector load to have any
effect. This is the same defect fixed for the Molmo v1 gate.

Refs #871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:core mlxcel-core: MLX FFI, primitives, KV cache, layers area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:high High priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant