[Model] Implement LoRA support for Qwen3ASRForConditionalGeneration

[petern48]

Purpose

This PR adds LoRA support for Qwen3ASRForConditionalGeneration model.

For this to work for the audio tower, I had to make a few additional changes:

• Implement get_num_mm_encoder_tokens()
• Replaced some nn.Linears with ReplicatedLinear along the audio tower path.
• Qwen3ASR seems to be our first model with a tower, but no connector. In gpu_model_runner.py, I found that the hasattr(self.model, "get_num_mm_connector_tokens") was improperly evaluating to True due to inheritance, despite the model not implementing get_num_mm_connector_tokens(). This was leading us to incorrectly go down that path and encounter an error. I've modified the condition to check if connector actually exists in the mapping.

Fixes #37223

Test Plan

I tested on the following public adaptor available on HuggingFace: ha0yuan/Qwen3-ASR-LoRa-ChineseAviation-Tiny.

vllm serve Qwen/Qwen3-ASR-1.7B \
  --enable-lora \
  --enable-tower-connector-lora \
  --lora-modules aviation=ha0yuan/Qwen3-ASR-LoRa-ChineseAviation-Tiny \
  --port 8000

I also double-checked that the adapters are properly shown when querying the /v1/models endpoint:

curl localhost:8000/v1/models | jq .

{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3-ASR-1.7B",
      "object": "model",
      ...
      "root": "Qwen/Qwen3-ASR-1.7B",
      ...
    },
    {
      "id": "medpl",
       ...
      "root": "AleksanderObuchowski/Qwen3-ASR-1.7B-med-pl-lora-decoder-only",
      ...
    }
}

Then I used a Python script to load in a .wav file and query the /v1/chat/completions endpoint. Specifically, I used this audio file as input.

Test Result

Before this PR, the server would error with
ValueError: Qwen3ASRForConditionalGeneration does not support LoRA yet.

After this change, the server starts up properly, and I successfully queried the /v1/chat/completions endpoint.

Querying both the raw model and the adaptor, I verified that the output differs when the LoRa adaptor is enabled. The outputs are below. (Notice, the raw model transcribes to numbers (e.g 9 and 10), while the adaptor transcribes the numbers to words ("nine" and "ten).

== no LoRA ==
language English<asr_text>November the 10th, Wednesday, 9 p.m. I'm standing in a dark alley. After waiting several hours, the time has come. A woman with long dark hair approaches. I have to act, and fast, before she realizes what has happened. I must find out.

== with LoRA ==
language English<asr_text>November the tenth, Wednesday, nine p.m. I'm standing in a dark alley. After waiting several hours, the time has come. A woman with long dark hair approaches. I have to act, and fast, before she realizes what has happened. I must find out.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request adds LoRA support for the Qwen3ASRForConditionalGeneration model. The changes introduce the SupportsLoRA interface and define the necessary attributes (packed_modules_mapping, embedding_modules, lora_skip_prefixes) to correctly apply LoRA adapters to the language model portion of the model, while skipping the audio tower. My review of these changes did not identify any issues.

[mergify]

Documentation preview: https://vllm--37247.org.readthedocs.build/en/37247/

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[jeejeelee]

Have you tested this PR with the real LoRA adapter?

[petern48]

@jeejeelee Yes, I have. I listed the exact public adapter I used in the PR description. I also just linked the data I used and the text output it generated, verifying that querying the adapter successfully leads to output that is different from the raw base model.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @petern48.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jeejeelee]

Please fix the pre-commit failure, thank you

[petern48]

Error: PR must have the 'ready' label or the author must have at least 4 merged PRs (found 0).

@jeejeelee I think you just need to add the ready label. #37544 introduced a change that prevents pre-commit from running by default. I have the pre-commit setup locally, so it should already be formatted correctly. It was also already passing before, and all I've done since then is merge with main. Could you add it, please?