Skip to content

Make DeepCompile ZeRO-3 scheduling memory-aware#8129

Draft
tohtana wants to merge 18 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix/deepcompile-zero3-memory-aware-scheduler-review-fixes
Draft

Make DeepCompile ZeRO-3 scheduling memory-aware#8129
tohtana wants to merge 18 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix/deepcompile-zero3-memory-aware-scheduler-review-fixes

Conversation

@tohtana

@tohtana tohtana commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

This PR improves DeepCompile's scheduling and robustness.

  • Make DeepCompile ZeRO-3 scheduling memory-aware by deriving a data-parallel-group-consistent gather budget from conservative per-node memory profiles and using it to order parameter gathers and releases.
  • Preserve ZeRO parameter ownership and compiled-backward lifecycle state across eager fallback, compilation failures, and overlapping DeepSpeed engines.
  • Add rank-consistency diagnostics, focused regression coverage, and documentation explaining the scheduling model and its safety constraints.

tohtana and others added 18 commits July 2, 2026 13:47
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant