[ET-VK][custom_ops] Standardize test case label format across binaries#19570
Merged
Conversation
Previously the test case labels printed by each custom_ops test binary used wildly inconsistent formats. Examples from a recent on-device pass:
ACCU conv1d_dw [1,16,64] K=3 s=1 p=1 d=1 Tex(HP) f32
ACCU I=144 Buf(4W)
correctness_1_64_32_Buffer_Float
small_1d_linear_weight
ACCU In=1,16,4,4 r=2 same_qp [4W4C->4W4C]
ACCU 32->3 I=64,64 g=1 k=3 Buf(4C1W) [general]
Standardize all 21 binaries on:
[ACCU/PERF] [INPUT->OUTPUT dtype] [shape data + inline op params] [Storage(layout)] [optional suffix]
Concrete examples after this commit:
ACCU f32->f32 [1,16,64] k3 s1 p1 d1 Tex(HP)
ACCU i8->i8 [144]+[144] Buf(4W)
ACCU f32->f32 [1,64]x[32,64] Buf(WP)
ACCU i32->f32 [1,8] lw Tex(WP)
ACCU i8->i8 [1,16,4,4] r=2 Buf(4W4C)->Buf(4W4C) [same_qp]
ACCU f32->f32 [1,3,64,64]x[32,3,3,3] s1 p1 d1 g1 Buf(4C1W) [general]
Adds three helpers to test/custom_ops/utils.h:
- dtype_short(vkapi::ScalarType) -> "f32" / "f16" / "i8" / "i32"
- shape_bracket(std::vector<int64_t>) -> "[N,C,H,W]" form
- make_test_label(prefix, in_dtype, out_dtype, shape_str, storage_str, suffix) assembles the four sections with two-space separators and an optional bracketed suffix.
Each binary's TestCase-builder helper now invokes make_test_label rather than rolling its own ostringstream. Op-specific shape detail (kernel/stride/padding/dilation/groups for conv, ratio for pixel_shuffle, output_padding for transposed conv, layout-transition arrows for clone/qdq) is constructed inline since it varies per op. Information that was load-bearing in the old labels (impl_selector tags, +bias/no_bias flags, same_qp/diff_qp markers, const_b for binary ops, multi-stage Buf->Buf transitions) is preserved as a bracketed suffix.
Two minor format deviations worth flagging:
- choose_qparams_per_row produces two outputs (scale tensor + zero_point tensor). Encoded as out_dtype = "f32,i8" -- a comma-separated tuple after the arrow -- rather than forcing a single token.
- test_embedding_q4gsw loses the free-form prose labels (small_1d_linear_weight, llama_3_2_1b_prefill_*). Replaced with structural shape info + an lw / scales-dtype suffix; readers can still derive the same context but it is less greppable as a "story".
Differential Revision: [D105059941](https://our.internmc.facebook.com/intern/diff/D105059941/)
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19570
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 4 Pending, 2 Unrelated Failures, 1 Unclassified FailureAs of commit 0d3c455 with merge base 1992bdd ( UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:
FLAKY - The following job failed but was likely due to flakiness present on trunk:
BROKEN TRUNK - The following job failed but was present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This was referenced May 13, 2026
GregoryComer
approved these changes
May 13, 2026
48d488b
into
gh/SS-JIA/534/base
172 of 185 checks passed
SS-JIA
pushed a commit
that referenced
this pull request
May 14, 2026
Previously the test case labels printed by each custom_ops test binary used wildly inconsistent formats. Examples from a recent on-device pass:
ACCU conv1d_dw [1,16,64] K=3 s=1 p=1 d=1 Tex(HP) f32
ACCU I=144 Buf(4W)
correctness_1_64_32_Buffer_Float
small_1d_linear_weight
ACCU In=1,16,4,4 r=2 same_qp [4W4C->4W4C]
ACCU 32->3 I=64,64 g=1 k=3 Buf(4C1W) [general]
Standardize all 21 binaries on:
[ACCU/PERF] [INPUT->OUTPUT dtype] [shape data + inline op params] [Storage(layout)] [optional suffix]
Concrete examples after this commit:
ACCU f32->f32 [1,16,64] k3 s1 p1 d1 Tex(HP)
ACCU i8->i8 [144]+[144] Buf(4W)
ACCU f32->f32 [1,64]x[32,64] Buf(WP)
ACCU i32->f32 [1,8] lw Tex(WP)
ACCU i8->i8 [1,16,4,4] r=2 Buf(4W4C)->Buf(4W4C) [same_qp]
ACCU f32->f32 [1,3,64,64]x[32,3,3,3] s1 p1 d1 g1 Buf(4C1W) [general]
Adds three helpers to test/custom_ops/utils.h:
- dtype_short(vkapi::ScalarType) -> "f32" / "f16" / "i8" / "i32"
- shape_bracket(std::vector<int64_t>) -> "[N,C,H,W]" form
- make_test_label(prefix, in_dtype, out_dtype, shape_str, storage_str, suffix) assembles the four sections with two-space separators and an optional bracketed suffix.
Each binary's TestCase-builder helper now invokes make_test_label rather than rolling its own ostringstream. Op-specific shape detail (kernel/stride/padding/dilation/groups for conv, ratio for pixel_shuffle, output_padding for transposed conv, layout-transition arrows for clone/qdq) is constructed inline since it varies per op. Information that was load-bearing in the old labels (impl_selector tags, +bias/no_bias flags, same_qp/diff_qp markers, const_b for binary ops, multi-stage Buf->Buf transitions) is preserved as a bracketed suffix.
Two minor format deviations worth flagging:
- choose_qparams_per_row produces two outputs (scale tensor + zero_point tensor). Encoded as out_dtype = "f32,i8" -- a comma-separated tuple after the arrow -- rather than forcing a single token.
- test_embedding_q4gsw loses the free-form prose labels (small_1d_linear_weight, llama_3_2_1b_prefill_*). Replaced with structural shape info + an lw / scales-dtype suffix; readers can still derive the same context but it is less greppable as a "story".
Differential Revision: [D105059941](https://our.internmc.facebook.com/intern/diff/D105059941/)
ghstack-source-id: 381655474
Pull Request resolved: #19570
SS-JIA
pushed a commit
that referenced
this pull request
May 14, 2026
Previously the test case labels printed by each custom_ops test binary used wildly inconsistent formats. Examples from a recent on-device pass:
ACCU conv1d_dw [1,16,64] K=3 s=1 p=1 d=1 Tex(HP) f32
ACCU I=144 Buf(4W)
correctness_1_64_32_Buffer_Float
small_1d_linear_weight
ACCU In=1,16,4,4 r=2 same_qp [4W4C->4W4C]
ACCU 32->3 I=64,64 g=1 k=3 Buf(4C1W) [general]
Standardize all 21 binaries on:
[ACCU/PERF] [INPUT->OUTPUT dtype] [shape data + inline op params] [Storage(layout)] [optional suffix]
Concrete examples after this commit:
ACCU f32->f32 [1,16,64] k3 s1 p1 d1 Tex(HP)
ACCU i8->i8 [144]+[144] Buf(4W)
ACCU f32->f32 [1,64]x[32,64] Buf(WP)
ACCU i32->f32 [1,8] lw Tex(WP)
ACCU i8->i8 [1,16,4,4] r=2 Buf(4W4C)->Buf(4W4C) [same_qp]
ACCU f32->f32 [1,3,64,64]x[32,3,3,3] s1 p1 d1 g1 Buf(4C1W) [general]
Adds three helpers to test/custom_ops/utils.h:
- dtype_short(vkapi::ScalarType) -> "f32" / "f16" / "i8" / "i32"
- shape_bracket(std::vector<int64_t>) -> "[N,C,H,W]" form
- make_test_label(prefix, in_dtype, out_dtype, shape_str, storage_str, suffix) assembles the four sections with two-space separators and an optional bracketed suffix.
Each binary's TestCase-builder helper now invokes make_test_label rather than rolling its own ostringstream. Op-specific shape detail (kernel/stride/padding/dilation/groups for conv, ratio for pixel_shuffle, output_padding for transposed conv, layout-transition arrows for clone/qdq) is constructed inline since it varies per op. Information that was load-bearing in the old labels (impl_selector tags, +bias/no_bias flags, same_qp/diff_qp markers, const_b for binary ops, multi-stage Buf->Buf transitions) is preserved as a bracketed suffix.
Two minor format deviations worth flagging:
- choose_qparams_per_row produces two outputs (scale tensor + zero_point tensor). Encoded as out_dtype = "f32,i8" -- a comma-separated tuple after the arrow -- rather than forcing a single token.
- test_embedding_q4gsw loses the free-form prose labels (small_1d_linear_weight, llama_3_2_1b_prefill_*). Replaced with structural shape info + an lw / scales-dtype suffix; readers can still derive the same context but it is less greppable as a "story".
Differential Revision: [D105059941](https://our.internmc.facebook.com/intern/diff/D105059941/)
ghstack-source-id: 381655474
Pull Request resolved: #19570
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Previously the test case labels printed by each custom_ops test binary used wildly inconsistent formats. Examples from a recent on-device pass:
Standardize all 21 binaries on:
Concrete examples after this commit:
Adds three helpers to test/custom_ops/utils.h:
Each binary's TestCase-builder helper now invokes make_test_label rather than rolling its own ostringstream. Op-specific shape detail (kernel/stride/padding/dilation/groups for conv, ratio for pixel_shuffle, output_padding for transposed conv, layout-transition arrows for clone/qdq) is constructed inline since it varies per op. Information that was load-bearing in the old labels (impl_selector tags, +bias/no_bias flags, same_qp/diff_qp markers, const_b for binary ops, multi-stage Buf->Buf transitions) is preserved as a bracketed suffix.
Two minor format deviations worth flagging:
Differential Revision: D105059941