[AIMIGRAPHX-885][AIMIGRAPHX-987] Use External Stream Contexts by TedThemistokleous · Pull Request #4775 · ROCm/AMDMIGraphX

TedThemistokleous · 2026-04-10T16:12:05Z

Motivation

Customer workload seeing some stalls during inference. This allows us to use the customer hipSteam passed to context via run_async so that we don't need to internally sync and manage a thread within MIGraphX. This allows the synchronization to be handled externally.

As an added benefit if not external thread is used we should fall back to the old fork_join run on the GPU where we internally create a stream to sync events onto.

Technical Details

Adds additional conditions to the wait_for , finish_on calls in context.cpp such that we avoid new stream creation for async runs while also simplifying much of the code.

Test cases have been added for this to ensure we don't break existing functionality.
Additional code added to ensure we set external libraries like BLAS and MIOPEN to use the default stream on clear

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

- Added: New functionality.
- Changed: Changes to existing functionality.
- Removed: Functionality or support that has been removed. (Compared to a previous release)
- Optimized: Component performance that has been optimized or improved.
- Resolved Issues: Known issues from a previous version that have been resolved.
- Not Applicable: This PR is not to be included in the changelog.

codecov · 2026-04-10T18:11:15Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #4775      +/-   ##
===========================================
- Coverage    92.88%   92.65%   -0.23%     
===========================================
  Files          587      587              
  Lines        30348    30356       +8     
===========================================
- Hits         28187    28124      -63     
- Misses        2161     2232      +71

Files with missing lines	Coverage Δ
src/include/migraphx/context.hpp	`81.48% <100.00%> (+12.99%)`	⬆️
src/program.cpp	`81.70% <100.00%> (+0.54%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Enables MIGraphX GPU async execution to run directly on a caller-provided HIP stream (external stream contexts) to reduce internal synchronization/stalls, and adds GPU tests to validate external-stream behavior and fallback behavior.

Changes:

Add external-stream support in gpu::context/hip_device::stream (override stream used by the context during async eval).
Adjust async synchronization logic (wait_for / finish_on) to avoid creating/using an extra internal stream when an external stream is provided.
Add a comprehensive new GPU test suite covering external stream override, async eval behavior, and fallback paths.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
`src/targets/gpu/include/migraphx/gpu/context.hpp`	Adds external stream override plumbing and modifies async sync behavior to use caller stream.
`test/gpu/external_stream.cpp`	Adds new GPU tests for external stream override, async eval correctness, and state cleanup expectations.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

bdevorem

couple questions, thanks Ted

bdevorem

I think a rebase/merge will solve the CI problems. lgtm otherwise

pfultz2

I dont think we should do this in the wait_for and finish_on functions as it could change the semantics of the function. Instead we should add a use_queue method to the context interface and use that directly.

TedThemistokleous · 2026-04-16T21:09:42Z

I think a rebase/merge will solve the CI problems. lgtm otherwise

Sure rebased this off develop

TedThemistokleous · 2026-04-17T03:57:34Z

I dont think we should do this in the wait_for and finish_on functions as it could change the semantics of the function. Instead we should add a use_queue method to the context interface and use that directly

Okay let me add this. This is similar and just do the create/set in the use_queue or use_external() thread?

I dont think we should do this in the wait_for and finish_on functions as it could change the semantics of the function. Instead we should add a use_queue method to the context interface and use that directly.

So is the idea then run_async() -> bind the stream if its not the null/default stream? otherwise we just create an internal stream for regular run()?

causten · 2026-04-29T21:02:39Z

Test	Batch	New Rate (3315f1)	Old Rate (`315edf8`)*	Diff	Status
torchvision-resnet50	64	2,147.80	3,171.06	-32.27%	🔴
torchvision-resnet50_fp16	64	4,547.13	6,692.36	-32.05%	🔴
torchvision-densenet121	32	1,515.92	2,712.65	-44.12%	🔴
torchvision-densenet121_fp16	32	4,434.40	4,557.05	-2.69%	✅
torchvision-inceptionv3	32	1,722.32	1,800.54	-4.34%	✅
torchvision-inceptionv3_fp16	32	2,833.11	2,826.78	0.22%	✅
cadene-inceptionv4	16	803.71	827.46	-2.87%	✅
cadene-resnext64x4	16	758.34	787.20	-3.67%	✅
slim-mobilenet	64	8,442.08	8,457.30	-0.18%	✅
slim-nasnetalarge	64	208.17	229.14	-9.15%	🔴
slim-resnet50v2	64	1,602.42	3,384.74	-52.66%	🔴
bert-mrpc-onnx	8	335.49	1,159.92	-71.08%	🔴
bert-mrpc-tf	1	137.59	485.90	-71.68%	🔴
pytorch-examples-wlang-gru	1	321.50	409.66	-21.52%	🔴
pytorch-examples-wlang-lstm	1	63.32	463.65	-86.34%	🔴
torchvision-resnet50_1	1	87.96	757.02	-88.38%	🔴
cadene-dpn92_1	1	79.88	444.46	-82.03%	🔴
cadene-resnext101_1	1	230.05	365.58	-37.07%	🔴
onnx-taau-downsample	1	220.86	399.93	-44.78%	🔴
dlrm-criteoterabyte	1	32.24	32.62	-1.19%	✅
dlrm-criteoterabyte_fp16	1	48.59	52.69	-7.78%	🔴
agentmodel	1	9,914.44	8,390.95	18.16%	🔆
unet_fp16	2	46.56	57.35	-18.81%	🔴
resnet50v1_fp16	1	428.71	943.09	-54.54%	🔴
resnet50v1_int8	1	391.18	918.19	-57.40%	🔴
bert_base_cased_fp16	64	1,055.33	1,104.15	-4.42%	✅
bert_large_uncased_fp16	32	310.71	348.13	-10.75%	🔴
bert_large_fp16	1	205.43	205.25	0.09%	✅
distilgpt2_fp16	16	2,102.58	2,101.53	0.05%	✅
yolov5s	1	573.70	566.41	1.29%	✅
tinyllama	1	45.45	46.07	-1.35%	✅
vicuna-fastchat	1	43.60	44.11	-1.16%	✅
whisper-tiny-encoder	1	419.81	421.39	-0.38%	✅
whisper-tiny-decoder	1	414.44	418.27	-0.92%	✅
llama2_7b	1	19.67	20.49	-3.99%	✅
qwen1.5-7b	1	23.67	23.68	-0.01%	✅
phi3-3.8b	1	25.98	26.88	-3.37%	✅
llama3-8b	1	20.20	21.89	-7.73%	🔴
whisper-large-encoder	1	9.26	10.33	-10.33%	🔴
whisper-large-decoder	1	98.66	103.61	-4.78%	✅
mistral-7b	1	20.63	23.92	-13.77%	🔴
FLUX.1-schnell	1	768.85	759.98	1.17%	✅

Regressions detected 🔴

* No develop baseline was found for this PR's branch point; compared against the latest available develop run instead.

causten · 2026-04-29T21:02:40Z

Test	Status	Result
bert-mrpc-onnx	✅	PASSED: MIGraphX meets tolerance
bert-mrpc-tf	❌	ERROR - check error output traceback Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 313, in main import tensorflow as tf File "/usr/local/lib/python3.10/dist-packages/tensorflow/init.py", line 38, in from tensorflow.python.tools import module_util as _module_util File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/init.py", line 36, in from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 26, in self_check.preload_check() File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/platform/self_check.py", line 63, in preload_check from tensorflow.python.platform import _pywrap_cpu_feature_guard ImportError: libamdhip64.so.6: cannot open shared object file: No such file or directory
pytorch-examples-wlang-gru	✅	PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm	✅	PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte	✅	PASSED: MIGraphX meets tolerance
agentmodel	✅	PASSED: MIGraphX meets tolerance
unet	✅	PASSED: MIGraphX meets tolerance
resnet50v1	✅	PASSED: MIGraphX meets tolerance
bert_base_cased_fp16	✅	PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16	🔴	FAILED: MIGraphX is not within tolerance - check verbose output
bert_large	✅	PASSED: MIGraphX meets tolerance
yolov5s	✅	PASSED: MIGraphX meets tolerance
tinyllama	✅	PASSED: MIGraphX meets tolerance
vicuna-fastchat	✅	PASSED: MIGraphX meets tolerance
whisper-tiny-encoder	✅	PASSED: MIGraphX meets tolerance
whisper-tiny-decoder	✅	PASSED: MIGraphX meets tolerance
distilgpt2_fp16	✅	PASSED: MIGraphX meets tolerance
llama2_7b	✅	PASSED: MIGraphX meets tolerance
qwen1.5-7b	✅	PASSED: MIGraphX meets tolerance
phi3-3.8b	✅	PASSED: MIGraphX meets tolerance
llama3-8b	✅	PASSED: MIGraphX meets tolerance
whisper-large-encoder	❌	ERROR - check error output traceback Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 224, in main model = migraphx.parse_onnx(model_name, default_dim_value=batch) RuntimeError: /data/src/include/migraphx/op/convolution.hpp:102: normalize_compute_shape: CONVOLUTION: mismatched channel numbers
whisper-large-decoder	✅	PASSED: MIGraphX meets tolerance
mistral-7b	✅	PASSED: MIGraphX meets tolerance
FLUX.1-schnell	✅	PASSED: MIGraphX meets tolerance

TedThemistokleous · 2026-04-29T22:37:26Z

Added changes based on Paul's comments so that we odn't modify wait_for, finish_on and just use a use_queue and set_queue_context.

CharlieL7

There's the tools/include/context.hpp file that looks like the file that generates the type erasure interface. The use_queue interface should probably be generated from there.

pfultz2

a

Can easily check has_value() as nullptr means default stream here.

bdevorem · 2026-05-21T22:26:08Z

is the codecov report up to date? might want to handle that before merging too

…tr and non null stream

…sues with typing.

pfultz2 · 2026-05-22T13:20:42Z

+        // save time?" (which may itself be empty, meaning no binding).  A
+        // single-level optional cannot distinguish "no save" from "saved
+        // empty binding" now that external_stream is itself an optional.
+        std::optional<std::optional<hipStream_t>> previous_stream{};


Remove previous_stream. The previous stream is always s.

Avoid repeat in setting raw stream handles for rocblas/miopen

…ce we've removed set_external_stream. Use set_queue instead

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

TedThemistokleous self-assigned this Apr 10, 2026

TedThemistokleous added the Perf Improve label Apr 10, 2026

TedThemistokleous requested a review from causten as a code owner April 10, 2026 16:12

TedThemistokleous mentioned this pull request Apr 10, 2026

[AIMIGRAPHX-885] Clean up pad/slice Copies to use external stream and cleanup memory alloc/dealloc ROCm/onnxruntime#233

Merged

TedThemistokleous requested a review from kahmed10 April 10, 2026 16:14

causten requested a review from bdevorem April 10, 2026 21:27

bdevorem requested a review from Copilot April 15, 2026 14:56

Copilot started reviewing on behalf of bdevorem April 15, 2026 14:58 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

bdevorem reviewed Apr 15, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

Comment thread test/gpu/external_stream.cpp

TedThemistokleous force-pushed the use_external_contexts branch from d2cfb6b to 563c78b Compare April 16, 2026 15:22

TedThemistokleous requested a review from bdevorem April 16, 2026 20:04

bdevorem approved these changes Apr 16, 2026

View reviewed changes

pfultz2 requested changes Apr 16, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

TedThemistokleous force-pushed the use_external_contexts branch from 7b92f18 to a164acb Compare April 17, 2026 03:55

TedThemistokleous mentioned this pull request Apr 17, 2026

Rocm7.2.3 changes #4796

Merged

TedThemistokleous changed the title ~~[AIMIGRAPHX-885] Use External Stream Contexts~~ [AIMIGRAPHX-885][AIMIGRAPGX-987] Use External Stream Contexts Apr 24, 2026

TedThemistokleous requested review from CharlieL7 and shivadbhavsar April 29, 2026 16:01

causten added the ok-to-test label Apr 29, 2026

TedThemistokleous requested a review from pfultz2 April 29, 2026 22:36

CharlieL7 requested changes Apr 30, 2026

View reviewed changes

TedThemistokleous requested a review from CharlieL7 April 30, 2026 19:59

pfultz2 reviewed May 6, 2026

View reviewed changes

TedThemistokleous added 3 commits May 21, 2026 16:43

Ensure async mode doesn't use wait_for/finish_on syncs

d30fb3d

Fux event record

c9a9473

Use std::optional semantics for external_stream

3e7169a

Can easily check has_value() as nullptr means default stream here.

TedThemistokleous added 8 commits May 21, 2026 17:37

Fix optional assign with value_or vs value()

42f1001

Add clear external stream to handle case where we want to unbind

2a6bc0a

Update tests

d3107e9

Add change for coverage regarding context and program changes.

be90720

Update get_queue to mirror set_queue to preserve typing between nullp…

6f289e8

…tr and non null stream

fix format

098f78f

Update tests to use unsafe_get isntead of the lazy create to avoid is…

70939a9

…sues with typing.

Fix license

5ee9d1e

pfultz2 requested changes May 22, 2026

View reviewed changes

TedThemistokleous added 5 commits May 22, 2026 11:07

Create set_raw_stream() call

9c4f913

Avoid repeat in setting raw stream handles for rocblas/miopen

Remove previous steram, cleanup comments

adead17

Cleanup tests and wait() call

4fa9fde

Remove comment

0270739

Update license

816f9e1

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

pfultz2 reviewed May 22, 2026

View reviewed changes

Comment thread src/targets/gpu/include/migraphx/gpu/context.hpp Outdated

TedThemistokleous requested a review from pfultz2 May 22, 2026 19:00

Changes to context and handle format and outstanding test changes sin…

f0a9151

…ce we've removed set_external_stream. Use set_queue instead

TedThemistokleous force-pushed the use_external_contexts branch from 8c6d7c3 to f0a9151 Compare May 22, 2026 19:01

pfultz2 approved these changes May 22, 2026

View reviewed changes

pfultz2 and others added 2 commits May 22, 2026 14:48

Update src/targets/gpu/include/migraphx/gpu/context.hpp

3315f14

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Merge branch 'develop' into use_external_contexts

376b4d1

Conversation

TedThemistokleous commented Apr 10, 2026

Motivation

Technical Details

Changelog Category

Uh oh!

codecov Bot commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bdevorem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bdevorem left a comment

Choose a reason for hiding this comment

Uh oh!

pfultz2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TedThemistokleous commented Apr 16, 2026

Uh oh!

TedThemistokleous commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

causten commented Apr 29, 2026 • edited by gh-app-migraphx-bot-pr-write Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

causten commented Apr 29, 2026 • edited by gh-app-migraphx-bot-pr-write Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TedThemistokleous commented Apr 29, 2026

Uh oh!

CharlieL7 left a comment

Choose a reason for hiding this comment

Uh oh!

pfultz2 left a comment

Choose a reason for hiding this comment

Uh oh!

bdevorem commented May 21, 2026

Uh oh!

pfultz2 May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov Bot commented Apr 10, 2026 •

edited

Loading

TedThemistokleous commented Apr 17, 2026 •

edited

Loading

causten commented Apr 29, 2026 •

edited by gh-app-migraphx-bot-pr-write Bot

Loading

causten commented Apr 29, 2026 •

edited by gh-app-migraphx-bot-pr-write Bot

Loading

pfultz2 May 22, 2026 •

edited

Loading