Skip to content

Implement native clang launcher - #27401

Open
sbc100 wants to merge 1 commit into
emscripten-core:mainfrom
sbc100:native_clang_impl
Open

Implement native clang launcher#27401
sbc100 wants to merge 1 commit into
emscripten-core:mainfrom
sbc100:native_clang_impl

Conversation

@sbc100

@sbc100 sbc100 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Implement Phase 1 of the native launcher design document. Provides
high-performance native C++ compiler drivers (emcc, em++) using C++20
that directly invoke Clang for compile steps (-c, -S, -E) and seamlessly
fall back to python (emcc.py / em++.py) for link steps or complex
post-processing options.

This tool completely replaces both the windows-specific pylauncher and
the shell/bat entry points created by create_entry_points.py

These new binaries are currently optional since create_entry_points.py
can still be used instead.

Tested using embuilder build libc --force (compiling 1,075
files sequentially with EMCC_CORES=1, EMCC_USE_NINJA=0, and
EMCC_BATCH_BUILD=0). Note that this is likely an outlier since
its perfectly suited to demonstrating to speedup from this change (i.e.
lots of small source files and no linking).

Platform Before (Python Baseline) After (Native Launcher) Improvement Speedup
Linux 181.96 s (169.3 ms/file) 64.82 s (60.3 ms/file) -117.14 s (-109.0 ms/file) 2.81x
Windows 343.90 s (319.9 ms/file) 105.12 s (97.8 ms/file) -238.78 s (-222.1 ms/file) 3.27x
macOS 162.08 s (150.6 ms/file) 64.73 s (60.1 ms/file) -97.35 s (-90.5 ms/file) 2.50x

See: #26453

@sbc100
sbc100 marked this pull request as draft July 22, 2026 00:32
@sbc100 sbc100 changed the title [native] Implement native clang launcher frontend Implement native clang launcher frontend Jul 22, 2026
@sbc100 sbc100 changed the title Implement native clang launcher frontend Implement native clang launcher Jul 22, 2026
@sbc100
sbc100 force-pushed the native_clang_impl branch 6 times, most recently from 198f9c8 to c5fd0d6 Compare July 22, 2026 21:53
@sbc100
sbc100 marked this pull request as ready for review July 22, 2026 21:54
@sbc100
sbc100 force-pushed the native_clang_impl branch from c5fd0d6 to 3d058e6 Compare July 22, 2026 22:04
@sbc100
sbc100 requested a review from dschuff July 22, 2026 22:05
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 470205f to bac6b35 Compare July 22, 2026 23:22
@sbc100
sbc100 requested a review from brendandahl July 22, 2026 23:32
@dschuff

dschuff commented Jul 22, 2026

Copy link
Copy Markdown
Member

Here are some high level comments:

  1. Do we need response file support? Super long command lines are more common at link time, but the windows cmdline length limit is pretty low, so maybe a long path with a few include directives would be enough to push us over even at compile time.
  2. If we are using C++20, we could use std::string_view instead of std::string in a lot of places to avoid a lot of allocations and copies in string operations, e.g. is_emscripten_only_warning, is_dash_s_setting, and other places. I got a bunch of other similar suggestions from Gemini for how to reduce allocations and redundant filesystem operations that might be worth considering to reduce overhead on startup.

Comment thread tools/emcc_native/main.cpp Outdated
Comment thread tools/emcc_native/main.cpp Outdated
Comment thread tools/emcc_native/driver.cpp Outdated
Comment thread tools/emcc_native/driver.cpp Outdated
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 06b94ef to 9fec6c9 Compare July 23, 2026 00:22
@sbc100
sbc100 requested a review from dschuff July 23, 2026 00:23
@sbc100
sbc100 force-pushed the native_clang_impl branch from 9fec6c9 to b5f76dc Compare July 23, 2026 00:36
Comment thread tools/emcc_native/driver.cpp
Comment thread tools/emcc_native/driver.cpp
Comment thread tools/emcc_native/driver.cpp Outdated
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 9ac1ab1 to c94e316 Compare July 23, 2026 19:16
@sbc100
sbc100 force-pushed the native_clang_impl branch 3 times, most recently from 0abdc68 to b7f2e37 Compare July 23, 2026 22:37
@sbc100

sbc100 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Benchmark results for are in.. libc builds are much faster on all platforms! Yay

@sbc100
sbc100 requested review from dschuff, juj and kripken July 23, 2026 22:39
@sbc100

sbc100 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

I think once we land this and we start using it everywhere we can just remove the old shell/bat/ps1/pylauncher launchers.

Comment thread test/common.py
Comment thread tools/emcc_native/generated_settings.h Outdated

namespace emscripten {

inline const std::unordered_set<std::string> LINK_ONLY_FLAGS = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These can use std::string_view instead of std::string, and can be backed by the static C strings used in the initializers.
Gemini suggested even going further and using std::array instead of unordered_set, keeping them sorted, and use std::binary_search for the lookups. Not sure that's worth it though, it seems error-prone.

Comment thread tools/emcc_native/tests/test_native.cpp
Comment thread tools/emcc_native/CMakeLists.txt
sbc100 added a commit that referenced this pull request Jul 27, 2026
#27410)

I noticed this oversight while implementing the native launcher in
#27401
@sbc100
sbc100 force-pushed the native_clang_impl branch from bc1e09d to 90c38f6 Compare July 27, 2026 19:49
sbc100 added a commit to sbc100/emscripten that referenced this pull request Jul 27, 2026
…ig file

This ended up breaking emsdk in ways that would require more complexity
to fix.  See emscripten-core/emsdk#1752.

This issue is that if emsdk is installed in the path contains spaces
the NODE_JS path, for example, then needs to be quoted correctly
in the config file.  e.g. `NODE_JS = '"path with spaces/bin/node"'.

Instead I think we should just revert this change.  It turns out
not to be needed for emscripten-core#27401 anyway since the LLVM_ROOT is never
a list anyway.
sbc100 added a commit that referenced this pull request Jul 27, 2026
This ended up breaking emsdk in ways that would require more complexity
to fix. See emscripten-core/emsdk#1752.

This issue is that if emsdk is installed in the path contains spaces the
NODE_JS path, for example, then needs to be quoted correctly in the
config file. e.g. `NODE_JS = '"path with spaces/bin/node"'.

Instead I think we should just revert this change. It turns out not to
be needed for #27401 anyway since the LLVM_ROOT is never a list anyway.
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 0fe001d to 3fd0a72 Compare July 28, 2026 20:46
Implement Phase 1 of the native launcher design document. Provides
high-performance native C++ compiler drivers (emcc, em++) using C++20
that directly invoke Clang for compile steps (-c, -S, -E) and seamlessly
fall back to python (emcc.py / em++.py) for link steps or complex
post-processing options.

Output executables are placed in ./bin using CMake and Ninja. Adds CI
matrix testing for Linux, macOS, and Windows.

These new binaries are currently 100% optional and the python versions
can still be used without building them.

Tested using `embuilder build libc --force` on Linux CI (compiling 1,075 files sequentially with `EMCC_CORES=1`, `EMCC_USE_NINJA=0`, and `EMCC_BATCH_BUILD=0`):

| Metric | Before (Python Baseline) | After (Native Launcher) | Improvement |
| :--- | :---: | :---: | :---: |
| **Total Time** | **181.96 s** (3m 2s) | **64.82 s** (1m 5s) | **-117.14 s (64.4% faster)** |
| **Time per File** | 169.3 ms | 60.3 ms | **-109.0 ms / invocation** |
| **Speedup Factor** | 1.0x | **2.81x** | — |

**Key Takeaways:**

* **64.4% Reduction in Build Time:** Bypassing Python interpreter startup
  shaves off **~109.0 ms of overhead per compiler invocation** on Linux CI
  (a **2.81x speedup**).
* **Windows Impact:** Because process spawning and `python.exe` startup
  carry significantly higher overhead on Windows than on Linux, the
  percentage speedup on Windows CI is expected to be even larger.

See: emscripten-core#26453
@sbc100
sbc100 force-pushed the native_clang_impl branch from 3fd0a72 to b142bbf Compare July 29, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants