Skip to content

fix(llama-cpp): explain tensor count mismatch - #11597

Open
localai-org-maint-bot wants to merge 2 commits into
masterfrom
fix/gguf-tensor-count-diagnostic
Open

fix(llama-cpp): explain tensor count mismatch#11597
localai-org-maint-bot wants to merge 2 commits into
masterfrom
fix/gguf-tensor-count-diagnostic

Conversation

@localai-org-maint-bot

Copy link
Copy Markdown
Collaborator

Description

This PR preserves llama.cpp tensor-count load errors and appends an actionable hint that the GGUF layout may be incompatible with the selected backend or the file may be corrupt. It leaves unrelated load errors unchanged.

Ollama-published monolithic Gemma 4 blobs use Ollama's private compatibility layer to hide embedded projector tensors. LocalAI uses upstream llama.cpp, so its loader sees the extra tensors and rejects that internal blob layout. The model-scoped backend process then exits after failed initialization; LocalAI itself remains available.

Closes #11593

Verification

g++ -std=c++17 backend/cpp/llama-cpp/model_load_error_test.cpp -o /tmp/localai-model-load-error-test
/tmp/localai-model-load-error-test
git diff --check

The standalone regression test covers the exact expected 2131, got 720 diagnostic, a generic tensor-count mismatch, and an unrelated load error.

Notes for Reviewers

The change deliberately does not import Ollama's compatibility patches, pre-parse GGUF metadata, or alter RPC status and backend lifecycle.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

llama.cpp reports the same tensor-count error for unsupported model layouts and damaged GGUF files. Add a focused hint so operators can update the backend or verify the model without losing the upstream diagnostic.

Assisted-by: Codex:gpt-5.6
The generated gRPC source tree omitted the new header and test. Every llama.cpp-derived backend therefore failed when grpc-server.cpp included the missing header.

Assisted-by: Codex:gpt-5.6 [systematic-debugging]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cuda13-llama-cpp backend crashes loading gemma4 GGUF: "wrong number of tensors; expected 2131, got 720"

1 participant