fix: SDXL ControlNet (diffusers naming + graph size)

[fszontagh]

Summary

Fixes SDXL ControlNet, which has never worked end-to-end in this repo. Loading any diffusers-format SDXL ControlNet (thibaud's OpenPoseXL2.safetensors, Stability's controlnet-*-sdxl-1.0) against an SDXL base failed with 844 back-to-back "not in model metadata" errors during validation, one per tensor. Two independent gaps are involved and both need fixing:

1. name_conversion.cpp: convert_tensor_name only routes diffusers→LDM names through convert_diffusers_unet_to_original_sdxl when the tensor starts with "model.diffusion_model.". ControlNet tensors have no such prefix (they arrive as down_blocks.…, controlnet_down_blocks.…, controlnet_cond_embedding.…, add_embedding.…, time_embedding.…, controlnet_mid_block.…, conv_in.…), so the converter was never called and the LDM block map declared in control.hpp (input_blocks.*, zero_convs.*, input_hint_block.*, middle_block_out.*) had zero overlap with what the file provided.
2. control.hpp: CONTROL_NET_GRAPH_SIZE = 1536 was sized for the SD 1.5 ControlNet (transformer_depth={1,1,1}). SDXL ControlNet's transformer_depth={1,2,10} blows past that on the first compute call with GGML_ASSERT(cgraph->n_nodes < cgraph->size) failed - a second, latent bug that (2) would have exposed as soon as (1) was fixed.

SD 1.5 ControlNet is unaffected because those files (control_v11p_sd15_*) ship in LDM naming directly and stay under the old graph budget.

Related Issue / Discussion

Closes long-standing #172 ("Are SDXL Controlnet models supported?") and #661 ("Stable Diffusion XL Control Net support"). Note #661's specific ask about xinsir/controlnet-union-sdxl-1.0 is not covered here - that model is the ControlNet-Union variant (extra control_type_proj embeds + Union routing), a different architecture, and would be a separate PR.

Additional Information

The fix

src/name_conversion.cpp

• New convert_diffusers_controlnet_to_original_sdxl(name) that first calls the existing UNet SDXL converter (which already handles down_blocks, mid_block, time_embedding, add_embedding, conv_in and their resnet/attention internals), then applies the ControlNet-only mappings:
  • controlnet_cond_embedding.conv_in.* → input_hint_block.0.*
  • controlnet_cond_embedding.blocks.<k>.* → input_hint_block.<2k+2>.* for k=0..5
  • controlnet_cond_embedding.conv_out.* → input_hint_block.14.*
  • controlnet_down_blocks.<i>.* → zero_convs.<i>.0.*
  • controlnet_mid_block.* → middle_block_out.0.*
• New is_diffusers_controlnet_name(name) recognizes the top-level heads used by diffusers ControlNet checkpoints.
• Wired in the // controlnet section of convert_tensor_name, gated on sd_version_is_sdxl(version) so SD 1.5 goes through the existing pre-normalized path unchanged and other architectures aren't touched.

src/model/diffusion/control.hpp

• CONTROL_NET_GRAPH_SIZE bumped from the hardcoded 1536 to MAX_GRAPH_SIZE (the same budget the main UNet uses). The graph node array cost is tiny (~2.6 MiB of pointer slots, alloc-time only, not per-step) and matches what the SDXL UNet already reserves.

Verified against the file

OpenPoseXL2.safetensors header parsed directly: 844 keys, top-level namespaces before the fix are all diffusers with zero overlap with the LDM block map:

top-level	count	mapped LDM equivalent
down_blocks	572	input_blocks.<3i+j+1>.…
mid_block	226	middle_block.<0/1/2>.…
controlnet_down_blocks	18	zero_convs.<i>.0.…
controlnet_cond_embedding	16	input_hint_block.<even>.…
add_embedding	4	label_emb.0.<0/2>.… (SDXL micro-cond)
time_embedding	4	time_embed.<0/2>.…
controlnet_mid_block	2	middle_block_out.0.…
conv_in	2	input_blocks.0.0.…

After the fix, the load prints zero "not in model metadata" errors and generation runs to completion.

End-to-end test

./build/bin/sd-cli \
  -m /data/SD_MODELS/checkpoints/SDXL/AcornXL_V2.safetensors \
  --control-net /data/SD_MODELS/controlnet/OpenPoseXL2.safetensors \
  --control-image pose.png --control-strength 0.9 \
  -p "1boy, knight, walking, side view" -n "text, 1girl" \
  --cfg-scale 6 --sampling-method dpm++2m --scheduler karras \
  -H 1024 -W 1024 --steps 20 --seed 42 \
  --offload-to-cpu --max-vram -1 --stream-layers \
  -o out.png

Result: knight in side view, walking, matching the OpenPose skeleton. Generation completes in 175s on a 12 GiB card via CPU streaming (see docs/performance.md for the streaming flags).

Not in scope

• ControlLoRA (control-lora-*-rank256.safetensors): different architecture (LoRA-decomposed .down/.up deltas), separate PR.
• ControlNet Union / Promax (controlnet-union-sdxl-1.0_promax.safetensors): extra control_type_proj embeds and Union routing logic, separate PR.

Checklist

• I have read and confirmed this PR follows the contribution guidelines.