Skip to content

feat: hybrid lora - #264

Open
kevssim wants to merge 21 commits into
modelscope:mainfrom
kevssim:feat/spectral-hybrid-lora
Open

feat: hybrid lora#264
kevssim wants to merge 21 commits into
modelscope:mainfrom
kevssim:feat/spectral-hybrid-lora

Conversation

@kevssim

@kevssim kevssim commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

  • Add Spectral Hybrid LoRA training support for the Transformers backend.
  • Support selecting Hybrid mode through train_mode: hybrid.
  • Add Hybrid LoRA server and cookbook configurations.

experiment

方法 finqa-acc tatqa-acc tatqa-f1 mmlu-acc
Qwen3.5-9B 0 0 0 0.866
FFT(lr=1.25e-6) 0.458 0.589 0.694 0.795
LoRA(lr=2.5e-5) 0.456 0.572 0.674 0.850
Hybrid-LoRA(lr=2.5e-5) 0.715 0.780 0.861 0.836

loss curve

image

@kevssim kevssim changed the title Feat/spectral hybrid lora Feat/hybrid lora Aug 13, 2026
@kevssim kevssim changed the title Feat/hybrid lora feat: hybrid lora Aug 14, 2026
@kevssim
kevssim marked this pull request as ready for review August 18, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant