[AMDGPU] asyncmark/wait.asyncmark intrinsics produce incorrect code for 3-stage software-pipelined loops

[jerryyin]

The llvm.amdgcn.asyncmark and llvm.amdgcn.wait.asyncmark intrinsics (introduced in #180467) produce numerically incorrect results when used in 3-stage software-pipelined loops with buffer_load_dwordx4 ... lds (async DMA to LDS) on gfx950. 2-stage pipelines are unaffected.

Reproducer

Given the .optimized.ll from a 4096×4096×4096 f32 GEMM with 3-stage pipelining, replacing the asyncmark intrinsics with explicit s_waitcnt calls produces correct results while the original IR produces ~40% wrong output elements.

The replacement:

; REMOVED:
call void @llvm.amdgcn.asyncmark()

; REPLACED:
call void @llvm.amdgcn.wait.asyncmark(i16 2)
→ call void @llvm.amdgcn.s.waitcnt(i32 8184)  ; vmcnt(8) = 2 groups × 4 loads/group

call void @llvm.amdgcn.wait.asyncmark(i16 0)
→ call void @llvm.amdgcn.s.waitcnt(i32 8176)  ; vmcnt(0)

Both versions compiled with llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx950 -O3. The LLVM IR and assembly for both the broken (original) and working (fixed) cases are attached.

A self-contained reproducer is available at: https://github.com/jerryyin/scripts/tree/master/iree/reproducers/asyncmark_bug

Minimal reproduction from assembly (reproduce_from_asm.sh): assembles both checked-in .s files to HSACOs, substitutes each into an otherwise-identical IREE vmfb (same compilation, same dispatch, same inputs), and compares against a baseline. The only variable is which .hsaco is loaded. Use this for reproducing.

Results

Case	max abs diff vs baseline	Wrong elements	Verdict
Original (with asyncmark)	~50	~6.8M / 16.8M (40%)	FAIL
Modified (asyncmark → s_waitcnt)	0.000000	0 / 16.8M	PASS

Consistently reproducible across runs. The exact wrong values vary between runs (characteristic of a race condition), but the failure rate is consistently ~40%.

What We Know

• 922 instructions in both, identical instruction count
• Identical s_waitcnt values: both produce vmcnt(8) and vmcnt(0) at the same positions — mergeAsyncMarks() computes the correct values
• Identical register allocation and loop structure
• Identical kernel descriptors (LDS size, VGPRs, SGPRs)
• The only difference: 4 ALU instructions reordered in the prolog (v_bfe_u32, v_and_b32, v_lshlrev_b32, s_mov_b32 — all independent address computations)

The asyncmark intrinsics carry IntrHasSideEffects, which makes them scheduling barriers in the machine scheduler DAG (isGlobalMemoryObject() returns true). Removing them changes the scheduler's freedom in the prolog, producing a slightly different instruction ordering. Despite the reordering involving only data-independent ALU instructions, one ordering consistently produces correct results and the other consistently produces ~40% wrong elements on gfx950.

The failure pattern (non-deterministic wrong values, consistent ~40% failure rate) is characteristic of a hardware timing-dependent race condition in the async DMA-to-LDS path, triggered by the specific instruction ordering in the prolog (wrong values, consistent rate).

[llvmbot]

@llvm/issue-subscribers-backend-amdgpu

Author: Zhuoran Yin (jerryyin)

Details

The `llvm.amdgcn.asyncmark` and `llvm.amdgcn.wait.asyncmark` intrinsics (introduced in #180467) produce numerically incorrect results when used in 3-stage software-pipelined loops with `buffer_load_dwordx4 ... lds` (async DMA to LDS). 2-stage pipelines are unaffected.

Reproducer

Given the .optimized.ll from a 4096×4096×4096 f32 GEMM with 3-stage pipelining, replacing the asyncmark intrinsics with explicit s_waitcnt calls produces correct results while the original IR produces ~40% wrong output elements.

The replacement:

; REMOVED:
call void @<!-- -->llvm.amdgcn.asyncmark()

; REPLACED:
call void @<!-- -->llvm.amdgcn.wait.asyncmark(i16 2)
→ call void @<!-- -->llvm.amdgcn.s.waitcnt(i32 8184)  ; vmcnt(8) = 2 groups × 4 loads/group

call void @<!-- -->llvm.amdgcn.wait.asyncmark(i16 0)
→ call void @<!-- -->llvm.amdgcn.s.waitcnt(i32 8176)  ; vmcnt(0)

Both versions compiled with llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx950 -O3. The LLVM IR and assembly for both the broken (original) and working (fixed) cases are attached.

A self-contained reproducer script is available at: https://github.com/jerryyin/scripts/tree/master/iree/reproducers/asyncmark_bug

Results

Case	max abs diff vs baseline	Wrong elements	Verdict
Original (with asyncmark)	~50	~6.8M / 16.8M (40%)	FAIL
Modified (asyncmark → s_waitcnt)	0.000000	0 / 16.8M	PASS

Analysis

The ASYNCMARK/WAIT_ASYNCMARK pseudo-instructions inherit IntrHasSideEffects from their intrinsic definitions in IntrinsicsAMDGPU.td. This has two consequences:

1. Scheduling barrier: SIInstrInfo::isGlobalMemoryObject() (via TargetInstrInfo::isGlobalMemoryObject()) returns true for instructions with unmodeled side effects. This makes the pseudos act as global memory barriers in the machine scheduler DAG, producing structurally different instruction ordering and register allocation.

2. Score tracking: In SIInsertWaitcnts.cpp, async LDS DMA writes are excluded from normal LDSDMA tracking (isNonAsyncLdsDmaWrite check at line 1206) and routed through AsyncScore/AsyncMarks instead. The mergeAsyncMarks() function incorrectly merges scores during loop backedge processing when 3+ marks interact simultaneously.

The assembly diff between original (822 lines) and fixed (817 lines) shows different register allocation and instruction ordering — confirming that the scheduling barrier effect, not just vmcnt values, contributes to the incorrect codegen.

[ssahasra]

@jerryyin wow it's cool that you already have an analysis of the potential issue! My hands are full right now, but I will make sure some progress is made either by me, or someone else in the compiler team. I will definitely make time for this on the coming Monday, IST. Meanwhile, if you find a way to resolve this, please don't hesitate, and do let me know.

[jerryyin]

@ssahasra Thanks Sameer. I don't have a good enough LLVM understanding to be honest. Actually I've pushed LLM far enough that it is able to populate a commit that fixed my issues (when tested locally). I think you are the right person decide whether this is the right direction - and correct it if it isn't. I'm not in a rush on this though. Knowing that you'd make time for this in the coming Monday is good enough for me timeline wise. Thank you!

[jerryyin]

I'm producing a second script to reproduce: reproduce_from_asm. If the previous reproducing instructions are not clear enough please use this new script. The only "external binary" are iree-compile and iree-run-module. It clearly shows the few lines of assembly difference is essential to correctness.

Full build instructions:

# 1. Clone and init submodules
git clone https://github.com/iree-org/iree.git
cd iree
git submodule update --init

# 2. Configure
cmake -G Ninja -B build -S . \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_C_COMPILER=clang \
    -DCMAKE_CXX_COMPILER=clang++ \
    -DIREE_BUILD_COMPILER=ON \
    -DIREE_BUILD_BUNDLED_LLVM=ON \
    -DIREE_TARGET_BACKEND_ROCM=ON \
    -DIREE_HAL_DRIVER_HIP=ON \
    -DIREE_ROCM_TEST_TARGET_CHIP=gfx950

# 3. Build
cmake --build build --target iree-compile iree-run-module llvm-mc lld

# 4. Edit reproduce_from_asm.sh paths, then run
./reproduce_from_asm.sh

If it helps tracking I can start a jira ticket for this.

[adeshcom14]

Hi @jerryyin,

I have done a detailed analysis of the asyncmark correctness issue you reported. I used the .ll files you provided (original/optimized.ll and fixed/optimized.ll), compiled them with local llc, and traced the full SIInsertWaitcnts pass. The findings differ from your bug analysis

1) SIInsertWaitcnts Debug Trace
I ran llc -debug-only=si-insert-waitcnts with -O3 enable on your original/optimized.ll and traced every step:

Prolog (bb.0): 2 groups of 4 DMA loads, each followed by asyncmark().

• After group 1: AsyncScore = 4, asyncmark() → AsyncMarks = [{4}], reset
• After group 2: AsyncScore = 8, asyncmark() → AsyncMarks = [{4}, {8}], reset
• End of prolog: ScoreUB = 8, ScoreLB = 0, AsyncMarks = [{4}, {8}]

Loop body (bb.1): 4 more DMA loads + asyncmark() + wait_asyncmark(2).

• After DMA loads: ScoreUB = 12, AsyncScore = 12
• asyncmark() → AsyncMarks = [{4}, {8}, {12}]
• wait_asyncmark(2) → looks up mark at index 0 (score 4):
• vmcnt = ScoreUB - MarkScore = 12 - 4 = 8 → s_waitcnt vmcnt(8)
• Oldest mark removed → AsyncMarks = [{8}, {12}]

Backedge merge (bb.1 → bb.1):

• Entry state ("Ours"): marks = {4, 8}, ScoreUB = 8, ScoreLB = 0
• Exit state ("Other"): marks = {8, 12}, ScoreUB = 12, ScoreLB = 4
• MyPending = 8, OtherPending = 8, NewUB = 8
• MyShift = 0, OtherShift = -4
• Most recent marks: max(8+0, 12-4) = max(8, 8) = 8
• Older marks: max(4+0, 8-4) = max(4, 4) = 4

The merge did not corrupt anything. The shift arithmetic correctly normalizes the absolute values. The loop body is not reprocessed. The mergeAsyncMarks() is working correctly.

2) Assembly Comparison: Original vs Fixed
By looking at the diff between the two assembly provided

• The s_waitcnt values are identical both produce vmcnt(8) in the loop body and vmcnt(0) at barrier points.

• The only difference is 4 ALU instructions reordered in the prolog, These instructions are independent integer address computations (v_lshlrev_b32, v_add_u32) with no data dependencies. The reordering has no data dependency on each other or on the DMA loads. Their results are consumed much later in the pipeline. The instruction ordering may have caused by the IntrHasSideEffects scheduling barrier. But we wanted the asyncmark to act as a barrier.

• Register allocation and instruction counts are identical.

I believe both produce functionally equivalent code with the same wait values.

3) And also Sameer PR #180467 already Tests 3-Stage Pipelines
The original PR 1 includes test cases in llvm/test/CodeGen/AMDGPU/asyncmark-pregfx12.ll that directly test 3-stage pipelined loops, including test_pipelined_loop (DMA-only) and test_pipelined_loop_with_global (mixed DMA + global loads).

I believe the above analysis is sufficient to demonstrate the issue that the asyncmark() tracking mechanism in upstream SIInsertWaitcnts.cpp. is working correctly.

[jerryyin]

@adeshcom14 Thanks for the detailed analysis. I agree with your trace — the s_waitcnt values are identical and mergeAsyncMarks() produces the correct vmcnt(8). I also agree the only assembly difference is 4 reordered ALU instructions in the prolog.

But the code produces wrong results on hardware. I've added a minimal reproducer (reproduce_from_asm.sh) that demonstrates this directly from the checked-in assembly files. It takes original/assembly.s and fixed/assembly.s, assembles each to an HSACO, substitutes each into an otherwise-identical IREE vmfb (same compilation flags, same dispatch, same inputs), and compares against a baseline:

original/assembly.s:  40.9% wrong elements, max_abs_diff = 51.5    FAIL
fixed/assembly.s:     0.0%  wrong elements, max_abs_diff = 0.0     PASS

Both assemblies go through llvm-mc → ld.lld → iree-compile --substitute → iree-run-module. The only variable is the .hsaco. The results are consistent across runs (non-deterministic in exact wrong values, but consistently ~40% wrong).

I'm not claiming mergeAsyncMarks() computes the wrong vmcnt. You're right that it doesn't. The issue is subtler — the asyncmark intrinsics, via IntrHasSideEffects, act as scheduling barriers that cause LLVM's machine scheduler to produce a different instruction ordering in the prolog. That different ordering, despite being data-independent, triggers a hardware correctness issue on gfx950 with async DMA-to-LDS. Removing the intrinsics (and replacing with explicit s_waitcnt) removes the scheduling barriers, producing a slightly different prolog ordering that doesn't trigger the issue.

To reproduce, build IREE from source and run the script above. I'd appreciate it if you could run this on a gfx950 machine to
confirm.

---

I now have more than one way to show this is a LLVM issue. The other approach I have is to just compile the same llvm ir with -O0 instead of -O3 and demonstrate the the wrong elements don't appear. Given that -O3 fails and -O0 fails this is a more straightforward proof that it is a backend bug.

# Compile original IR (with asyncmark intrinsics) at -O0
llc -mtriple=amdgcn-amd-amdhsa -mcpu=gfx950 -O0 -filetype=asm \
    original/optimized.ll -o original_O0.s

# Assemble → link → HSACO
llvm-mc -triple=amdgcn-amd-amdhsa -mcpu=gfx950 -filetype=obj \
    original_O0.s -o original_O0.o
ld.lld -shared original_O0.o -o original_O0.hsaco

# Substitute into IREE vmfb and run
iree-compile \
    --iree-hal-target-backends=rocm \
    --iree-rocm-target=gfx950 \
    --iree-llvmgpu-set-workgroup-distribution-along=x \
    --iree-llvmgpu-use-direct-load \
    --iree-llvmgpu-prefetch-num-stages=3 \
    --iree-hal-substitute-executable-object=matmul_dispatch_0=original_O0.hsaco \
    test_mm.mlir -o with_O0.vmfb

iree-run-module --module=with_O0.vmfb --device=hip \
    --input=@a.npy --input=@b.npy --output=@out_O0.npy

Results:

           | asyncmark intrinsics | opt level | s_waitcnt values       | verdict
-----------+----------------------+-----------+------------------------+---------
llc -O3    | yes                  | -O3       | vmcnt(8), vmcnt(0)     | FAIL (40% wrong)
llc -O3    | no (explicit waitcnt)| -O3       | vmcnt(8), vmcnt(0)     | PASS
llc -O0    | yes                  | -O0       | vmcnt(8), vmcnt(0)     | PASS

All three produce identical s_waitcnt vmcnt(8) and s_waitcnt vmcnt(0). The only case that fails is asyncmark intrinsics + -O3, where the machine scheduler reorders prolog instructions around the IntrHasSideEffects scheduling barriers. -O0 disables the machine scheduler, producing a different (correct) instruction ordering from the same IR.

[jerryyin]

Upon @krzysz00's reminder, I give you another reproducible without iree, at here.

• standalone reproduction script: This will compile and run a simple host harness that will compile the original assembly vs. fixed assembly
• driver.cpp A standalone host harness, invoking this 4kx4kx4k gemm. The result is compared with naive host gemm result
• Two assemblies, produced from llvm ir by swapping out async mark one in fixed folder another in original folder

Upon this you should have everything to play around: compile with -O0 and -O3, or use the manually updated async mark -> vmcnt llvmir. Hopefully there's no ambiguity right now. Let me know if you can't reproduce it.

[adeshcom14]

@jerryyin Thank you for providing the standalone reproducer and the -O0 vs -O3 comparison, that's been very helpful in narrowing down the investigation.

Unfortunately, I don't currently have access to a gfx950 machine. I have applied for Conductor access, but it will take some time to get approved. Once I have access, I'll run the reproducer, work on identifying the root cause, and follow up with my findings.

Thanks again for your help on this.

[krzysz00]

To relay some interesting investigation notes, if instead of going all the way to -O0, we just use the flags -O3 -enable-misched=false -enable-post-misched=false, this also fixes the correctness issue, which should hopefully produce a smaller surface to debug.

[adeshcom14]

I was able to reproduce the issue using your standalone reproduction script and driver.cpp. Also looks like the bug appears to be accidentally fixed by the commit 621fc87 — [AMDGPU] Implement LSR cost model for GFX9+ (#184138) by michaelselehov. Thanks @jerryyin for the reproducers and confirming that on latest upstream it is not reproducible.

The LSR cost model commit perturbed the induction variable structure and changed the instruction schedule, which is nothing but hiding the bug, not fixing the root cause.
However, the root cause is how the post RA scheduler is interacting with the asyncmark() intrinsic. The pre RA MI scheduler is correctly generating the output results containing asyncmark by keeping the post RA MI scheduler disabled. Thanks @krzysz00 for letting me know the flags.

The asyncmark pseudo-instruction emits only a comment no actual hardware instruction. I believe the post-RA scheduler is treating it as a real instruction and placed it between the below dependent instruction.

                              v_or_b32_e32 v12, 0x2000, v14
                              ; asyncmark     ----> should be a real instruction 
                              v_readfirstlane_b32 s3, v12

I have opened the ticket https://amd-hub.atlassian.net/browse/LCOMPILER-1681 for tracking purpose.