[TinyLoRA]tinylora implementation

[kashif]

Adds TinyLoRA, a new PEFT method based on "TinyLoRA: Learning to Reason in 13 Parameters". TinyLoRA achieves extreme parameter efficiency by replacing LoRA's trainable low-rank matrices with a tiny trainable vector projected through fixed random bases.

The key idea: given a frozen SVD decomposition W ≈ B @ A (where B = U @ sqrt(S) and A = sqrt(S) @ V^T), the weight update is delta_W = B @ R @ A where R is an r x r trainable matrix (following LoRA-XS). TinyLoRA takes this further by parameterizing R as a linear combination of fixed random projection matrices:

  R = sum_i(v[i] * P[i])

where v is the only trainable parameter (as small as 13 values) and P_i are fixed random matrices seeded deterministically.

Features

• Extreme efficiency: trainable parameter count is u per target module (or even less with weight tying), compared to r * (in + out) for LoRA
• Weight tying: configurable sharing of v vectors across layers via weight_tying (0.0 = no sharing, 1.0 = all layers share one v)
• SVD initialization: frozen A and B matrices computed from truncated SVD of pretrained weights, with singular values distributed equally via sqrt(S)
• Full layer support: nn.Linear, Conv1D, and nn.Embedding
• Merge/unmerge: full support including safe merge with NaN checking
• LoRA conversion: supports_lora_conversion() -> True — delta weights can be converted to standard LoRA format via get_delta_weight
• Deterministic projections: P matrices are seeded per-layer for reproducibility; optionally saved in checkpoints (save_projection=True)

Config

  from peft import TinyLoraConfig, get_peft_model

  config = TinyLoraConfig(
      r=2,              # SVD rank (frozen)
      u=64,             # trainable vector dimension
      weight_tying=0.0, # 0.0=no sharing, 1.0=full sharing
      target_modules="all-linear",
  )
  model = get_peft_model(base_model, config)

Architecture

• TinyLoraLayer (base): SVD decomposition, projection init, get_delta_weight, supports_lora_conversion
• Linear / Embedding: forward pass, merge/unmerge
• TinyLoraModel: weight tying groups, shared v parameter management via nested ModuleDict/ParameterDict
• update_layer follows LoRA's config-object pattern: (adapter_name, tinylora_v, v_key, r, config, **kwargs)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kashif]

cc @jxmorris12 I have an implementation of TinyLoRA if you can kindly have a look?

[githubnemo]

Hey @kashif :)

Thanks for the PR, this is already solid.
Merging with main should hopefully resolve the CI errors.

Some questions and comments below.

[kashif]

thanks fixing

[kashif]

@githubnemo should be ready for another review thanks

[githubnemo]

Thanks for the quick response :)

I think implementation-wise this is, except for two nits, good to go.

Let's add an example that showcases the primary use-case and add the method to the method comparison suite (maybe copy from method_comparison/MetaMathQA/experiments/lora/... and see where it takes us).

It'd also be stellar to have a commit message / PR description that is meaningful in the commit history.

[kashif]

ready @githubnemo

[BenjaminBossan]

Thanks for the PR Kashif. I ran the experiments on my machine and for got a test accuracy of 0% and 0.002% :)
This isn't really surprising with 3584 and 64 trainable parameters, respectively, and basically confirms the paper results. Still good to include them, but without context users could be drawing the wrong conclusion.

[kashif]

yes @BenjaminBossan i will test with the RL setup, we can wait if its ok, I want to also double check that nothing is wrong

[BenjaminBossan]

Out of curiosity, I wanted to check if TinyLoRA can achieve better scores if we increase the number of trainable parameters. So I took the default* setting and increased u and indeed, we can get decent results.

u	accelerator memory max	num trainable params	test_accuracy
512	20310917120	28672	0.30477634571645185
1024	20434649088	57344	0.3434420015163002
2024	20673724416	113344	0.31766489764973466

Given the still tiny number of trainable parameters, this result is quite respectable. This is also a nice confirmation that there is no major bug in the implementation.

I wonder if it would make sense to have a "maximalist" and a "minimalist" config, i.e. one with more trainable parameters and better score and one with extremely few trainable parameters (basically the current llama-3.2-3B-weight-tying) and low score.

*One more change I did was to increase r to 32 as 2 seemed pretty small to me, but that was just on a whim.

[githubnemo]

I wonder if it would make sense to have a "maximalist" and a "minimalist" config, i.e. one with more trainable parameters and better score and one with extremely few trainable parameters (basically the current llama-3.2-3B-weight-tying) and low score.

I think that's a good thing to have!

I also wondered if it would make sense to extend the target modules to all-linear since it is so cheap, but then again this would diverge from other experiments quite a lot and be not as comparable.

[kashif]

should we just document this? or add it somewhere else?

[BenjaminBossan]

I also wondered if it would make sense to extend the target modules to all-linear since it is so cheap, but then again this would diverge from other experiments quite a lot and be not as comparable.

I'd rather target either the attention xor the MLP part for consistence with other experiments.

should we just document this? or add it somewhere else?

What does "this" reference here?

[kashif]

ah sorry, i meant the minimal/maximal config?

[githubnemo]

ah sorry, i meant the minimal/maximal config?

Yes, let's add a 'maximalist' config with (possibly) r > 2 and u >= 2048. VeRA has ~128k parameters with 37.6% task accuracy according to https://huggingface.co/spaces/peft-internal-testing/PEFT-method-comparison - maybe it makes sense to match that setting (u ~= 2300) to see where the ceiling is within a reasonable memory budget (~+0.5GB).

[githubnemo]

The benchmarks spiraled a bit into side-projects. I think we can merge this without having the additional configs and check-in specific benchmarks at a later point in time.

Implementation LGTM.

[mengxi11]

tinylora实际性能可以达到论文里的多少？

[BenjaminBossan]

@mengxi11 We're working on a RL benchmark to check this: #3078.

[mengxi11]

再请问下关于tinylora的4bit量化微调训练貌似还没打通?

[BenjaminBossan]

TinyLoRA doesn't support quantization yet. We have plans for that (#3117) but that will probably take some time to land in PEFT. You could try something like

• LoRA + 4bit quantization + tiny rank (r=1)
• IA³ + 4bit quantization (set a high learning rate, ~1e-3)

Those are not the same as TinyLoRA, but they support bitsandbytes and are also very parameter efficient.