Compose: allowlisted backend environment variables for inference process

[VedantMadane]

Summary

docker model compose up can pass context-size, runtime-flags and speculative-decoding settings into ConfigureBackend, but it cannot pass process environment variables for the backend (for example GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 for AMD UMA workarounds).

The runner already supports this at the process layer: RunnerConfig.Env is validated by ValidateEnv and appended onto os.Environ() when spawning the backend. Compose and BackendConfiguration never expose a field for it, so compose users cannot set backend env without leaving the compose provider path.

Why this matters

• Use cases like llama.cpp UMA / unified memory need env vars on the inference process (see also Available unified memory in llama.cpp #994).
• Ollama-style compose snippets set env on the service container. DMR compose only configures the model backend and cannot mirror that today.
• Maintainers have already indicated env should not be exposed as an unauthenticated REST knob for security reasons. Compose is the right control plane because it is local provider configuration, not a broad network API.

Current code shape (main)

Layer	Status
pkg/inference/backends RunnerConfig.Env + ValidateEnv	Exists
inference.BackendConfiguration	No env field (only ContextSize, RuntimeFlags, speculative, backend-specific structs)
cmd/cli/commands/compose.go up	Flags: model, context-size, runtime-flags, backend, speculative-* only
compose metadata	Reflects those flags only

Proposed direction (for discussion)

1. Add an optional Environment []string (KEY=VALUE entries) to BackendConfiguration (or under LlamaCpp / shared block) and plumb it into RunnerConfig.Env when starting runners.
2. On docker model compose up, add a string-array flag such as --env KEY=VALUE (and/or a documented allowlist of keys).
3. Prefer an allowlist for keys (e.g. GGML_*, selected CUDA/Vulkan vars) so compose cannot inject arbitrary process env into the backend. Reject unknown keys with a clear error.
4. Advertise the parameter via compose metadata so Compose integration can document it.
5. Keep REST/OpenAI inference paths from accepting free-form env from remote clients.

Alternatives already available

• Manual llama-server after docker model pull (works, loses compose lifecycle).
• Global process env for the whole model-runner daemon (too coarse and easy to misconfigure).

Related

• Available unified memory in llama.cpp #994 (UMA / unified memory env example; this issue is the compose/API plumbing to set such vars safely)

I am happy to implement an allowlisted prototype if this direction sounds right.