[None][fix] Align perf launcher with pytest shard - #17134
Conversation
chienchunhung
changed the title
|
/bot run --disable-fail-fast |
|
PR_Github #63099 [ run ] triggered by Bot. Commit: |
Signed-off-by: Chien-Chun Hung <2679986+chienchunhung@users.noreply.github.com>
05880f6 to
7a2e0ec
Compare
|
/bot run --disable-fail-fast |
WalkthroughThe performance launcher now selects tests with pytest-split duration balancing. It validates split settings, requires one selected test, and reuses that test line for name and output-directory derivation. Tests cover shard selection and split-group mismatches. ChangesPytest-split test selection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Launcher
participant select_test_case_line
participant PytestCommand
participant DurationData
participant TestList
Launcher->>select_test_case_line: provide pytest command and split group
select_test_case_line->>PytestCommand: parse split configuration
select_test_case_line->>DurationData: load test durations
select_test_case_line->>TestList: read candidate test lines
select_test_case_line-->>Launcher: return selected test line
Launcher->>parse_test_case_name: pass selected test line
parse_test_case_name-->>Launcher: return test-case name
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
jenkins/scripts/perf/submit.py (2)
77-86: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider sharing the
pytestCommandline lookup.
_pytest_command_tokensandget_pytest_commands(Line 536) each locate theexport pytestCommand=line with their ownnext(...)expression. Extract one helper that returns the line. Both call sites then stay in sync if the prefix format changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/perf/submit.py` around lines 77 - 86, Extract the shared pytestCommand export-line lookup from _pytest_command_tokens and get_pytest_commands into a helper that returns the matching line or an empty value. Update both functions to use this helper, preserving their existing parsing and fallback behavior.
211-228: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider removing the second selection path.
mainalways passesselected_line, so thesplit_groupbranch at Lines 221-226 is unused in the CI flow. Two selection paths for one concept can diverge after a later change. Makeselected_linerequired, or have this fallback callselect_test_case_lineinstead of re-implementing positional selection.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@jenkins/scripts/perf/submit.py` around lines 211 - 228, The parse_test_case_name selection logic duplicates positional test selection even though main always supplies selected_line. Make selected_line required and remove the test_list_path, split_group, and fallback branch, or delegate fallback selection to select_test_case_line instead of indexing lines directly; preserve the existing selected-line parsing behavior.tests/unittest/scripts/test_perf_submit.py (1)
113-145: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest coverage summary
Changed test functions:
test_ci_submit_selects_same_least_duration_shard_as_pytest_split(added) — verifies least-duration group selection and the--durations-pathfallback tollm_src.test_ci_submit_rejects_split_group_disagreement(added) — verifies the--groupconsistency check.Test list registration: these are unit tests under
tests/unittest/scripts/, not integration tests, so no entry undertests/integration/test_lists/test-db/ortests/integration/test_lists/qa/is required. Confirm thattests/unittest/scripts/is already collected by an existing L0 unit-test stage.Verdict: sufficient for the primary fix, with these gaps worth closing:
- Equal durations. With
--splits 4and four tests, every group receives one test, so the tie-break order atsubmit.pyLine 155 is not exercised. Add a case with two identical durations.len(selected) != 1. Add a case where one group receives two tests, and assert the "requires exactly one test"ValueError.- Missing
--splits/--group. Add a case that asserts the positional fallback still returnslines[split_group - 1].As per path instructions: "Always produce a test coverage summary, even if no issues are found."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/scripts/test_perf_submit.py` around lines 113 - 145, Extend tests around select_test_case_line and test_ci_submit_selects_same_least_duration_shard_as_pytest_split to cover equal-duration tie-breaking, asserting the expected deterministic group selection. Add a case where one group contains multiple tests and assert the exact-one-test ValueError. Add a case without --splits or --group and verify positional fallback returns lines[split_group - 1]. Confirm the existing L0 unit-test collection includes tests/unittest/scripts/.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/scripts/perf/submit.py`:
- Around line 107-131: Update _load_pytest_split_durations so a missing
durations file is not silently converted to an empty dictionary: emit a clear
warning containing durations_path, or propagate an error when split_group > 0.
Preserve normal loading and validation for existing files, and ensure callers
can identify that duration-based selection is unavailable.
- Around line 134-165: Pin pytest-split to exactly version 0.10.0 in
requirements-dev.txt, preserving the existing dependency entry while adding the
explicit version constraint so CI uses the algorithm matched by
_select_least_duration_group.
---
Nitpick comments:
In `@jenkins/scripts/perf/submit.py`:
- Around line 77-86: Extract the shared pytestCommand export-line lookup from
_pytest_command_tokens and get_pytest_commands into a helper that returns the
matching line or an empty value. Update both functions to use this helper,
preserving their existing parsing and fallback behavior.
- Around line 211-228: The parse_test_case_name selection logic duplicates
positional test selection even though main always supplies selected_line. Make
selected_line required and remove the test_list_path, split_group, and fallback
branch, or delegate fallback selection to select_test_case_line instead of
indexing lines directly; preserve the existing selected-line parsing behavior.
In `@tests/unittest/scripts/test_perf_submit.py`:
- Around line 113-145: Extend tests around select_test_case_line and
test_ci_submit_selects_same_least_duration_shard_as_pytest_split to cover
equal-duration tie-breaking, asserting the expected deterministic group
selection. Add a case where one group contains multiple tests and assert the
exact-one-test ValueError. Add a case without --splits or --group and verify
positional fallback returns lines[split_group - 1]. Confirm the existing L0
unit-test collection includes tests/unittest/scripts/.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 272c70dc-8d52-4c8a-aac5-c830373da971
📒 Files selected for processing (2)
jenkins/scripts/perf/submit.pytests/unittest/scripts/test_perf_submit.py
| def _load_pytest_split_durations(tokens, llm_src): | ||
| durations_option = _pytest_option(tokens, "--durations-path") | ||
| if durations_option: | ||
| durations_path = durations_option | ||
| if not os.path.exists(durations_path): | ||
| durations_path = os.path.join( | ||
| llm_src, | ||
| "tests", | ||
| "integration", | ||
| "defs", | ||
| os.path.basename(durations_option), | ||
| ) | ||
| else: | ||
| durations_path = os.path.join(llm_src, "tests", "integration", "defs", ".test_durations") | ||
|
|
||
| try: | ||
| with open(durations_path, "r") as durations_file: | ||
| durations = json.load(durations_file) | ||
| except FileNotFoundError: | ||
| durations = {} | ||
| if isinstance(durations, list): | ||
| durations = dict(durations) | ||
| if not isinstance(durations, dict): | ||
| raise ValueError(f"Invalid pytest-split durations file: {durations_path}") | ||
| return durations, durations_path |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not fall back silently to empty durations.
If neither the --durations-path value nor the llm_src fallback exists, durations becomes {}. Every test then gets the same average weight of 1.0, and _select_least_duration_group degenerates to round-robin over the name-sorted list. pytest inside the container reads the real durations file, so it can select a different test for the same group. The launcher would then write the YAML config and the output directory of a test that pytest does not run.
Make the missing file visible. Print a warning at minimum, or raise when split_group > 0.
🛠️ Proposed change to surface the missing durations file
try:
with open(durations_path, "r") as durations_file:
durations = json.load(durations_file)
except FileNotFoundError:
+ print(
+ f"WARNING: pytest-split durations file not found: {durations_path}. "
+ "Shard selection may disagree with pytest."
+ )
durations = {}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _load_pytest_split_durations(tokens, llm_src): | |
| durations_option = _pytest_option(tokens, "--durations-path") | |
| if durations_option: | |
| durations_path = durations_option | |
| if not os.path.exists(durations_path): | |
| durations_path = os.path.join( | |
| llm_src, | |
| "tests", | |
| "integration", | |
| "defs", | |
| os.path.basename(durations_option), | |
| ) | |
| else: | |
| durations_path = os.path.join(llm_src, "tests", "integration", "defs", ".test_durations") | |
| try: | |
| with open(durations_path, "r") as durations_file: | |
| durations = json.load(durations_file) | |
| except FileNotFoundError: | |
| durations = {} | |
| if isinstance(durations, list): | |
| durations = dict(durations) | |
| if not isinstance(durations, dict): | |
| raise ValueError(f"Invalid pytest-split durations file: {durations_path}") | |
| return durations, durations_path | |
| def _load_pytest_split_durations(tokens, llm_src): | |
| durations_option = _pytest_option(tokens, "--durations-path") | |
| if durations_option: | |
| durations_path = durations_option | |
| if not os.path.exists(durations_path): | |
| durations_path = os.path.join( | |
| llm_src, | |
| "tests", | |
| "integration", | |
| "defs", | |
| os.path.basename(durations_option), | |
| ) | |
| else: | |
| durations_path = os.path.join(llm_src, "tests", "integration", "defs", ".test_durations") | |
| try: | |
| with open(durations_path, "r") as durations_file: | |
| durations = json.load(durations_file) | |
| except FileNotFoundError: | |
| print( | |
| f"WARNING: pytest-split durations file not found: {durations_path}. " | |
| "Shard selection may disagree with pytest." | |
| ) | |
| durations = {} | |
| if isinstance(durations, list): | |
| durations = dict(durations) | |
| if not isinstance(durations, dict): | |
| raise ValueError(f"Invalid pytest-split durations file: {durations_path}") | |
| return durations, durations_path |
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 122-122: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(durations_path, "r")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@jenkins/scripts/perf/submit.py` around lines 107 - 131, Update
_load_pytest_split_durations so a missing durations file is not silently
converted to an empty dictionary: emit a clear warning containing
durations_path, or propagate an error when split_group > 0. Preserve normal
loading and validation for existing files, and ensure callers can identify that
duration-based selection is unavailable.
| def _select_least_duration_group(lines, durations, splits, group): | ||
| """Mirror pytest-split's LeastDurationAlgorithm exactly.""" | ||
| if splits < 1: | ||
| raise ValueError(f"pytest --splits must be >= 1, got {splits}") | ||
| if group < 1 or group > splits: | ||
| raise ValueError(f"pytest --group must be in [1, {splits}], got {group}") | ||
|
|
||
| nodeids = [_test_nodeid(line) for line in lines] | ||
| relevant_durations = { | ||
| nodeid: float(durations[nodeid]) for nodeid in nodeids if nodeid in durations | ||
| } | ||
| average_duration = ( | ||
| sum(relevant_durations.values()) / len(relevant_durations) if relevant_durations else 1.0 | ||
| ) | ||
| items = [ | ||
| (line, nodeid, relevant_durations.get(nodeid, average_duration), original_index) | ||
| for original_index, (line, nodeid) in enumerate(zip(lines, nodeids)) | ||
| ] | ||
|
|
||
| if split_group > 0: | ||
| # pytest-split first sorts by item name, then performs a stable descending | ||
| # duration sort. It greedily places each item in the least-loaded group. | ||
| items.sort(key=lambda item: item[1]) | ||
| items.sort(key=lambda item: item[2], reverse=True) | ||
| selected = [[] for _ in range(splits)] | ||
| group_heap = [(0.0, group_index) for group_index in range(splits)] | ||
| heapq.heapify(group_heap) | ||
| for line, _nodeid, duration, original_index in items: | ||
| group_duration, group_index = heapq.heappop(group_heap) | ||
| selected[group_index].append((original_index, line)) | ||
| heapq.heappush(group_heap, (group_duration + duration, group_index)) | ||
|
|
||
| return [line for _original_index, line in sorted(selected[group - 1], key=lambda item: item[0])] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Find the pytest-split version constraint used by the CI test environment.
fd -t f -e txt -e toml -e cfg -e ini . requirements constraints 2>/dev/null | head -50
rg -n --hidden -g '!**/.git/**' 'pytest[-_]split' -C 2Repository: NVIDIA/TensorRT-LLM
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -u
printf '%s\n' '--- candidate dependency files ---'
git ls-files | rg '(^|/)(requirements[^/]*|constraints[^/]*|pyproject\.toml|setup\.cfg|tox\.ini|Pipfile|environment[^/]*|Dockerfile[^/]*)$|(^|/)\.github/workflows/|(^|/)jenkins/'
printf '%s\n' '--- pytest-split references ---'
rg -n --hidden -i -g '!**/.git/**' 'pytest[-_]split|pytest.split|least_duration|--splits|--group' .
printf '%s\n' '--- target file context ---'
sed -n '1,190p' jenkins/scripts/perf/submit.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 18239
🏁 Script executed:
#!/bin/bash
set -u
printf '%s\n' '--- dependency declarations ---'
cat -n requirements-dev.txt | sed -n '1,45p'
printf '%s\n' '--- related constraints and project metadata ---'
rg -n -C 3 'pytest[-_]split|pytest' requirements.txt constraints.txt pyproject.toml .github/workflows jenkins --glob '*.yml' --glob '*.yaml' --glob '*.groovy' --glob '*.sh' --glob '*.txt' --glob '*.toml'
printf '%s\n' '--- attribution entry ---'
sed -n '53495,53545p' ATTRIBUTIONS-Python.md
printf '%s\n' '--- focused tests ---'
sed -n '90,175p' tests/unittest/scripts/test_perf_submit.pyRepository: NVIDIA/TensorRT-LLM
Length of output: 35558
🌐 Web query:
pytest-split 0.10.0 LeastDurationAlgorithm source deterministic sort item name stable descending duration
💡 Result:
In pytest-split 0.10.0, the least_duration algorithm is designed to produce deterministic test splits, even when used with test-randomization plugins like pytest-randomly or pytest-random-order [1][2]. To ensure this determinism, the algorithm sorts items to create a consistent processing order regardless of the input order [3][2]. The process is as follows: 1. Indexing: Each test item is assigned its original index from the input list [3]. 2. Deterministic Sorting: The algorithm sorts the test items by their string representation (the item name) [3]. This step is explicitly implemented to ensure that the same set of tests is processed in the same order on all nodes, making the split deterministic [3][2]. 3. Duration-based Assignment: After the initial deterministic sort by name, the items are sorted by their duration in descending order (using reverse=True) [3]. These sorted items are then distributed among the available splits using a heap-based approach to minimize total group duration [3]. 4. Maintaining Order: Finally, the items assigned to each group are re-sorted by their original index (preserved from step 1) to ensure the relative order of tests is maintained within each group [3]. It is worth noting that while sorting by the string representation (name) was intended to ensure determinism, some users have noted that if multiple tests share the same name (e.g., in different modules) and lack distinct durations, this sorting approach might still result in non-deterministic splits [4]. Using nodeid is often cited as a more robust identifier for ensuring uniqueness in such cases [4]. [1][3][2][4]
Citations:
- 1: https://jerry-git.github.io/pytest-split/changelog/
- 2: Split deterministically regardless of test order Fix #23 jerry-git/pytest-split#52
- 3: https://jerry-git.github.io/pytest-split/api_docs/
- 4: fix least duration algo with random ordering jerry-git/pytest-split#109
🏁 Script executed:
#!/bin/bash
set -u
printf '%s\n' '--- CI installation of requirements-dev.txt ---'
rg -n -C 3 --hidden -g '!**/.git/**' 'requirements-dev\.txt|pip3? install|pip install' .github jenkins docker Makefile* build scripts 2>/dev/null | rg -i 'requirements-dev|pip3? install|requirements'
printf '%s\n' '--- all pytest-split declarations and generated metadata ---'
rg -n --hidden -g '!**/.git/**' '(^|[^A-Za-z0-9])pytest-split([^A-Za-z0-9]|$)|pytest_split' .Repository: NVIDIA/TensorRT-LLM
Length of output: 14002
Pin pytest-split to 0.10.0.
CI installs requirements-dev.txt without a version constraint. This selector matches pytest-split 0.10.0; an upgrade can change shard assignments.
🧰 Tools
🪛 Ruff (0.16.0)
[warning] 150-150: zip() without an explicit strict= parameter
Add explicit value for parameter strict=
(B905)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@jenkins/scripts/perf/submit.py` around lines 134 - 165, Pin pytest-split to
exactly version 0.10.0 in requirements-dev.txt, preserving the existing
dependency entry while adding the explicit version constraint so CI uses the
algorithm matched by _select_least_duration_group.
|
PR_Github #63099 [ run ] completed with state
|
2 participants
Summary
Root cause
The launcher interpreted
--split-group Nas the Nth line in the test list. Pytest instead applies pytest-split'sleast_durationalgorithm using.test_durations, so a shard can select a different test from the raw Nth line.That allowed the launcher to configure servers and the output directory for one test while pytest executed another test. During the NVBUG#6481034 investigation, this caused the targeted Kimi shard to write and collect artifacts under a different test identity.
Fix
Parse the pytest-split options from the generated
pytestCommand, load the same duration data, and mirror pytest-split 0.10.0'sLeastDurationAlgorithm. The launcher now validates that its group matches pytest's group and requires one selected test for each multi-node perf shard.This PR contains no transfer tracing, runtime product changes, KV-cache fraction changes, timeout changes, test configuration changes, or waiver changes.
Validation
pre-commit run --files jenkins/scripts/perf/submit.py tests/unittest/scripts/test_perf_submit.pypython -m py_compile jenkins/scripts/perf/submit.py tests/unittest/scripts/test_perf_submit.pypython -m pytest -q --confcutdir=tests/unittest/scripts tests/unittest/scripts/test_perf_submit.py— 12 passedRelated investigation: NVBUG#6481034 and #16918.
Dev Engineer Review
submit.pynow mirrors pytest-split 0.10.0 duration-balanced shard selection.selected_line=None.QA Engineer Review
test_ci_submit_selects_same_least_duration_shard_as_pytest_split.test_ci_submit_rejects_split_group_disagreement.test-db/orqa/files.