From 3c1b4de09fd74fd843a8f2a804e69001fd60c6e6 Mon Sep 17 00:00:00 2001 From: John Trujillo Date: Wed, 5 Aug 2026 14:02:26 -0500 Subject: [PATCH] feat(ai-assistant): warn before loading an oversized local model Selecting a .gguf now runs a pre-flight check before the path is persisted, which is what makes ai-core load it. The model's size and its declared shape are weighed against free RAM; if it looks too large the user is asked whether to go ahead, with Cancel as the default and the figures behind the judgement on screen. Declining leaves the previously selected model untouched. The check fails open throughout: an unreadable size, header or memory reading means no warning rather than a wrong one. - ModelMemoryEstimator splits mmap'd weights from the KV cache and compute buffers, sized from the model's own shape when the header provides it. - ModelMemoryGate decides Safe / Risky(TIGHT|INSUFFICIENT) / Unknown. - GgufHeaderReader reads only the metadata block, bounded by a byte budget, an entry cap and an array cap, and attributes shape keys to their own architecture so a multimodal file's clip.* values stay out. - UserConfirmation lets the load coroutine await the user's answer, so the flow stays one readable sequence across a rotation. ADFA-1798 --- ai-assistant/src/main/assets/docs/index.html | 22 ++ .../plugins/aiassistant/AiAssistantPlugin.kt | 37 +++ .../fragments/AiSettingsFragment.kt | 56 +++- .../fragments/MemoryWarningDialogFragment.kt | 153 +++++++++ .../aiassistant/memory/DeviceMemory.kt | 43 +++ .../memory/ModelMemoryEstimator.kt | 108 +++++++ .../aiassistant/memory/ModelMemoryGate.kt | 53 +++ .../plugins/aiassistant/util/ByteSize.kt | 25 ++ .../aiassistant/util/GgufHeaderReader.kt | 305 ++++++++++++++++++ .../aiassistant/util/ModelFileSource.kt | 132 ++++++++ .../viewmodel/AiSettingsViewModel.kt | 205 +++++++++--- .../aiassistant/viewmodel/UserConfirmation.kt | 68 ++++ ai-assistant/src/main/res/values/strings.xml | 8 + .../memory/ModelMemoryEstimatorTest.kt | 180 +++++++++++ .../aiassistant/memory/ModelMemoryGateTest.kt | 79 +++++ .../aiassistant/util/GgufHeaderReaderTest.kt | 283 ++++++++++++++++ .../plugins/aiassistant/util/GgufWriter.kt | 107 ++++++ .../AiSettingsViewModelMemoryTest.kt | 290 +++++++++++++++++ .../viewmodel/UserConfirmationTest.kt | 177 ++++++++++ .../plugins/aicore/LocalLlmBackend.kt | 5 + .../plugins/aicore/ModelLoadDiagnostics.kt | 22 ++ .../aicore/ModelLoadDiagnosticsTest.kt | 27 ++ 22 files changed, 2342 insertions(+), 43 deletions(-) create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/fragments/MemoryWarningDialogFragment.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/memory/DeviceMemory.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/memory/ModelMemoryEstimator.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/memory/ModelMemoryGate.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/util/ByteSize.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/util/GgufHeaderReader.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/util/ModelFileSource.kt create mode 100644 ai-assistant/src/main/kotlin/com/itsaky/androidide/plugins/aiassistant/viewmodel/UserConfirmation.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/memory/ModelMemoryEstimatorTest.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/memory/ModelMemoryGateTest.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/util/GgufHeaderReaderTest.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/util/GgufWriter.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/viewmodel/AiSettingsViewModelMemoryTest.kt create mode 100644 ai-assistant/src/test/kotlin/com/itsaky/androidide/plugins/aiassistant/viewmodel/UserConfirmationTest.kt diff --git a/ai-assistant/src/main/assets/docs/index.html b/ai-assistant/src/main/assets/docs/index.html index 3b69e634..e0799942 100644 --- a/ai-assistant/src/main/assets/docs/index.html +++ b/ai-assistant/src/main/assets/docs/index.html @@ -56,6 +56,28 @@

Choosing a backend

agent reads are sent to Google over HTTPS. +

Will this model fit in memory?

+

When you pick a local .gguf file, it is measured against the RAM + free on the device at that moment, before anything is loaded. If it looks too + large you get a warning with the actual figures and two choices:

+ +

The warning quotes two numbers. Memory to load is the weights: these are + memory-mapped, so they need not all fit at once — when they don't, the device pages + them in and out, which is why an oversized model can stall for minutes instead of + failing immediately. Memory to run is the KV cache and compute buffers, which + are ordinary allocations and do have to fit. That is why a model can be reported as + risky rather than impossible: the outcome genuinely depends on how much paging your + device will tolerate.

+

To fit a large model, close other apps and select it again, or pick a smaller or + more heavily quantized build. A Q4_K_M quantization of a 1–3B model is the most + likely to run comfortably.

+

What the agent can do