Skip to content

Guard DP dummy inputs around pending work#4738

Open
grimoire wants to merge 1 commit into
InternLM:mainfrom
grimoire:dp-forward-inputs-sleep-safe
Open

Guard DP dummy inputs around pending work#4738
grimoire wants to merge 1 commit into
InternLM:mainfrom
grimoire:dp-forward-inputs-sleep-safe

Conversation

@grimoire

@grimoire grimoire commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

This pr could reduce numbers of dummy inputs when dp>1 is enabled.

Copilot AI review requested due to automatic review settings July 7, 2026 08:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces unnecessary dummy forward inputs when data-parallel (DP) workers have real inputs pending in the CPU-side preprocess queue, improving DP forward efficiency and avoiding wasted dummy steps.

Changes:

  • Update DPForwardInputsMaker to treat _pre_in_que as pending real work, waiting briefly for preprocessing to complete before falling back to dummy inputs.
  • Add a small “yield to RPC delivery” window to reduce dummy forwards caused by event-loop/RPC scheduling gaps.
  • Add targeted unit tests covering the new DP input selection behavior; minor docstring punctuation normalization in guided spec tests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
lmdeploy/pytorch/engine/model_agent/inputs_maker.py Adds pending-real-input detection via _pre_in_que, preprocessing wait, and RPC-yield logic before dummy fallback.
lmdeploy/pytorch/engine/model_agent/agent.py Clarifies preprocess-loop contract: _pre_in_que represents pending real work until moved to _in_que.
tests/pytorch/engine/test_model_agent.py Adds unit tests verifying DPForwardInputsMaker.get() behavior across pending, RPC-yield, and sleeping scenarios.
tests/pytorch/spec_decode/test_guided_spec_integration.py Minor docstring character normalization (ASCII arrow/dash).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants