Fix pthreads normal-mutex debug implementation

[slowriot]

This PR fixes a regression in Emscripten's debug-only PTHREAD_MUTEX_NORMAL path that showed up under -pthread, and adds regression coverage for the two failure modes we reduced from the original application.

See #26619 for details on the regression.

Problem

In 4.0.11, Emscripten changed debug builds so PTHREAD_MUTEX_NORMAL no longer always takes the old fast a_cas(..., 0, EBUSY) path. The motivation was valid: in debug builds, a same-thread relock of a normal mutex should trap instead of hanging forever.

The problem is that this changed the observable semantics of the mutex lock word for normal mutexes in debug builds. In particular, normal mutexes started carrying owner-style state through the slow path instead of preserving the historical 0/EBUSY encoding.

That broke dlmalloc under -pthread debug system-library builds. dlmalloc uses normal pthread mutexes for its internal locks, and with the new debug path we were able to trigger allocator aborts with a minimal browser repro consisting only of:

• main thread allocation churn
• wasm worker allocation/deallocation churn
• -pthread
• -sWASM_WORKERS

No application logic, rendering, or complex containers were required.

The first part of the fix was therefore to restore historical _m_lock semantics for normal mutexes in debug Emscripten builds while keeping the new debug deadlock trapping behavior.

However, that exposed a second issue.

To preserve the debug deadlock check without using _m_lock, the intermediate fix stored normal-mutex ownership in _m_count. That was enough to fix the allocator regression, but it introduced a second false-positive deadlock under contention: _m_count is shared mutable state, so contending threads can observe stale owner information and incorrectly conclude that the current thread already owns the mutex.

This second issue showed up in the harness-facing repro as a deadlock assert during steady-state execution/reporting, even though the mutex was not actually being recursively acquired by the same thread.

Root Cause

There are really two related problems here:

1. Debug normal mutexes were no longer preserving the historical 0/EBUSY _m_lock encoding.
   That broke internal users such as dlmalloc.

2. _m_count is not a safe place to track debug-only ownership for normal mutexes under contention.
   It is shared state, so reading it from another thread is inherently racy and can produce stale-owner false positives.

So the correct design is:

• keep _m_lock semantics unchanged for normal mutexes
• keep debug deadlock detection
• do not put debug normal-mutex ownership into shared mutex fields that contending threads will race on

Fix

This PR implements that design.

Changes:

• system/lib/libc/musl/src/thread/pthread_mutex_trylock.c
• system/lib/libc/musl/src/thread/pthread_mutex_unlock.c
• system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c
• system/lib/libc/musl/src/internal/pthread_impl.h

What the fix does:

• For debug Emscripten builds, PTHREAD_MUTEX_NORMAL still keeps _m_lock in the historical 0/EBUSY form.
• Ownership for debug deadlock detection is tracked in a per-thread list anchored on struct pthread.
• On successful normal-mutex lock, the mutex is added to that thread's list.
• On normal-mutex unlock, it is removed from that thread's list.
• The debug deadlock assertion for normal mutexes checks that per-thread ownership list, not _m_count and not _m_lock.

Why this design:

• It preserves the intent of the 4.0.11 debug change: same-thread relock of a normal mutex still traps instead of silently hanging.
• It preserves the old _m_lock semantics that dlmalloc depended on.
• It avoids using shared mutable state for debug-only owner tracking, which was the source of the stale-owner false positives.
• It leaves non-normal mutex behavior alone.
• It leaves release/NDEBUG behavior alone.

A subtle but important implementation detail is that debug normal-mutex ownership is tracked in a per-thread list anchored on struct pthread, rather than in a separate _Thread_local variable or in shared mutex fields such as _m_lock / _m_count. This preserves the historical mutex lock-word semantics for normal mutexes, avoids racy shared ownership state, and also avoids introducing extra TLS state that broke other.test_threadprofiler during CI.

Why not simpler alternatives

I intentionally did not take these approaches:

• Reverting the 4.0.11 debug change entirely.
  That would discard the intended "trap instead of deadlock" behavior that motivated the change.

• Keeping _m_count as the owner field.
  That fixes the first failure but causes the second one because _m_count is shared and racy.

• Disabling assertions / forcing NDEBUG.
  That works around the issue but does not fix the debug runtime.

The patch here is meant to be a real upstream fix, not a workaround.

Tests

This PR adds two browser regression tests in test/test_browser.py:

• test/wasm_worker/pthread_mutex_debug_allocator_regression.cpp
  This captures the original allocator failure mode: debug -pthread + -sWASM_WORKERS plus trivial main/worker allocation traffic must stay clean.

• test/wasm_worker/pthread_mutex_debug_reporting_teardown.cpp
  This captures the stale-owner false-positive that appeared after the first fix was applied.

I also verified that the existing deadlock behavior is still intact via:

• other.test_pthread_mutex_deadlock
• other.test_threadprofiler

So the final state is:

• original allocator regression fixed
• follow-on stale-owner deadlock fixed
• intended debug deadlock trapping for real same-thread relock preserved
• existing --threadprofiler startup behavior preserved

Validation

The fix is validated with:

• browser.test_wasm_worker_pthread_mutex_debug_allocator_regression
• browser.test_wasm_worker_pthread_mutex_debug_reporting_teardown
• other.test_pthread_mutex_deadlock
• other.test_threadprofiler

[Copilot]

Pull request overview

This PR fixes a debug-only regression in Emscripten’s musl pthread normal-mutex path under -pthread (notably affecting dlmalloc with -sWASM_WORKERS) by restoring historical _m_lock semantics while keeping same-thread relock deadlock trapping via debug-only thread-local ownership tracking. It also adds browser regression tests covering the allocator abort and the subsequent stale-owner deadlock false-positive.

Changes:

• Restore PTHREAD_MUTEX_NORMAL debug behavior to keep _m_lock in the historical 0/EBUSY encoding while maintaining deadlock trapping.
• Add debug-only thread-local tracking of held normal mutexes using _m_prev/_m_next, and use it for deadlock assertions.
• Add two browser regression tests for the allocator regression and the reporting/teardown false-positive.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
system/lib/libc/musl/src/thread/pthread_mutex_trylock.c	Implements debug normal-mutex lock-word preservation and hooks normal-mutex ownership tracking.
system/lib/libc/musl/src/thread/pthread_mutex_unlock.c	Hooks normal-mutex unlock to remove from debug thread-local ownership tracking.
system/lib/libc/musl/src/thread/pthread_mutex_timedlock.c	Updates debug deadlock assertions to consult thread-local normal-mutex ownership list.
system/lib/libc/musl/src/internal/pthread_impl.h	Adds debug-only thread-local list + helpers for normal-mutex ownership tracking.
test/test_browser.py	Registers two new browser regression tests under the wasm worker suite.
test/wasm_worker/pthread_mutex_debug_allocator_regression.cpp	New browser repro focused on allocator churn under -pthread + wasm workers.
test/wasm_worker/pthread_mutex_debug_reporting_teardown.cpp	New browser repro focused on the follow-on stale-owner deadlock assert.
AUTHORS	Adds contributor entry.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.