[PAA+OPT] Layer normalization accuracy alignment and performance optimization by zhengshengning · Pull Request #78321 · PaddlePaddle/Paddle

zhengshengning · 2026-03-16T09:55:04Z

PR Category

Operator Mechanism

PR Types

Performance

Description

背景与动机

当前 Paddle 的 layer_norm 算子在 GPU 上使用的 kernel 与 PyTorch 的实现存在精度差异，主要体现在：

epsilon 参数类型为 float，在某些场景下精度不足
GPU kernel 中 mean/variance 的计算采用的数值方法与 PyTorch 不同

本 PR 通过引入精度兼容（accuracy-compatible）kernel，使 Paddle 的 layer_norm 在开启 FLAGS_use_accuracy_compatible_kernel 标志时，能够与 PyTorch 对齐精度。

主要改动

epsilon 参数精度提升：将 layer_norm 所有 kernel（CPU、GPU、XPU、OneDNN）及 InferMeta 中的 epsilon 参数类型从 float 改为 double，避免精度截断。
新增精度兼容 CUDA kernel：在 rms_norm_cuda_kernel.h 中新增 LayerNormFwdCompatKernel（前向）和 LayerNormBwdCompatKernel（反向），基于 Welford 在线算法实现数值稳定的 mean/variance 计算，与 PyTorch 的 layer_norm kernel 对齐。
精度兼容开关：通过 FLAGS_use_accuracy_compatible_kernel 全局标志控制是否启用精度兼容 kernel，默认关闭，不影响现有行为。
YAML 同步更新：ops.yaml 和 backward.yaml 中 layer_norm 的 epsilon 类型同步更新为 double。

是否引起精度变化

是。
FLAGS_use_accuracy_compatible_kernel打开时：
当 FLAGS_use_accuracy_compatible_kernel 开启时，layer_norm 的前向和反向计算将使用与 PyTorch 对齐的高精度 kernel，计算结果会有精度变化（更接近 PyTorch）。

FLAGS_use_accuracy_compatible_kernel默认关闭时：
默认FLAGS_use_accuracy_compatible_kernel关闭状态下，当layer_nrom不走高性能版本（FAST_LN_V1和FAST_LN_V2）时，且（!isPowerOfTwo(feature_size) && feature_size > 1024）时将会走与Torch精度对齐的分支（性能更好）。

paddle-bot · 2026-03-16T09:55:17Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

…atible kernels

… acc_layer_norm

codecov-commenter · 2026-03-19T21:08:33Z

Codecov Report

❌ Patch coverage is 92.40506% with 6 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@53b56df). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...addle/fluid/ir_adaptor/translator/op_translator.cc	87.09%	4 Missing ⚠️
.../core/distributed/auto_parallel/inferspmd_utils.cc	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop   #78321   +/-   ##
==========================================
  Coverage           ?   92.40%           
==========================================
  Files              ?       13           
  Lines              ?       79           
  Branches           ?        0           
==========================================
  Hits               ?       73           
  Misses             ?        6           
  Partials           ?        0

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

wanghuancoder

LGTM

lugimzzz

lgtm

DrRyanHuang

有速度对比数据嘛，有的话辛苦也贴一下~

Sign in to view

-                          ? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))
-                          : 1e-5f;
+    const double eps = op_desc.HasAttr("epsilon")
+                           ? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))


DrRyanHuang · 2026-03-23T04:03:12Z

-                          ? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))
-                          : 1e-5f;
+    const double eps = op_desc.HasAttr("epsilon")
+                           ? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))


Suggested change

? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))

? PADDLE_GET_CONST(double, op_desc.GetAttr("epsilon"))

DrRyanHuang · 2026-03-23T04:03:58Z

    int begin_norm_axis =
        PADDLE_GET_CONST(int, layer_norm_op->Op()->GetAttr("begin_norm_axis"));
-    float eps =
+    double eps =


下面的 PADDLE_GET_CONST 也改成 double

wanghuancoder

LGTM

From00

LGTM

…mization (PaddlePaddle#78321)

…rformance optimization #78321 (#78490) * [PAA+OPT] Layer normalization accuracy alignment and performance optimization * fix * fix2

[PAA] Align layer_norm precision with PyTorch by adding accuracy-comp…

e0d27dd

…atible kernels

zhengshengning force-pushed the acc_layer_norm branch from 6d43903 to e0d27dd Compare March 16, 2026 11:51

fix

7ef2871

zhengshengning force-pushed the acc_layer_norm branch from 8b4d0ce to 7ef2871 Compare March 16, 2026 12:39

zhengshengning added 2 commits March 17, 2026 11:50

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f510007

… acc_layer_norm

add LayerNormOpTranscriber

0168e25

zhengshengning requested review from ForFishes, From00 and LiYuRio as code owners March 17, 2026 08:30

rm fix of ops_exclude.yaml

87854d2

zhengshengning force-pushed the acc_layer_norm branch from 8895c5e to 87854d2 Compare March 17, 2026 09:01

fix

f3e8cd8

zhengshengning force-pushed the acc_layer_norm branch from fe36951 to f3e8cd8 Compare March 17, 2026 10:37

add PD_SPECIALIZE_InferSpmdFnCallHelper_FOR_ATTRIBUTE

17cd33b

zhengshengning force-pushed the acc_layer_norm branch from 7ace83c to 17cd33b Compare March 17, 2026 13:45

zhengshengning added 3 commits March 18, 2026 11:27

fix LayerNormFusePass

7ab10b2

fix

76dd65a

add PADDLE_WITH_CUDA in layer_norm_kernel

4c6a917

zhengshengning requested review from JiabinYang and xiaoguoguo626807 as code owners March 18, 2026 12:59

zhengshengning removed request for ForFishes, From00, JiabinYang, LiYuRio and xiaoguoguo626807 March 18, 2026 13:00

fix

8eeaa25

zhengshengning changed the title ~~[PAA] Align layer_norm precision with PyTorch by adding accuracy-compatible kernels~~ [PAA+OPT] Layer normalization accuracy alignment and performance optimization Mar 19, 2026

opt layer_norm

2533d5e

wanghuancoder previously approved these changes Mar 20, 2026

View reviewed changes

lugimzzz approved these changes Mar 20, 2026

View reviewed changes

DrRyanHuang reviewed Mar 23, 2026

View reviewed changes

fix

6fca1fd

zhengshengning dismissed wanghuancoder’s stale review via 6fca1fd March 23, 2026 08:51

wanghuancoder approved these changes Mar 25, 2026

View reviewed changes

xiaoguoguo626807 approved these changes Mar 25, 2026

View reviewed changes

From00 approved these changes Mar 25, 2026

View reviewed changes

Jiang-Jia-Jun approved these changes Mar 25, 2026

View reviewed changes

swgu98 added the skip-ci: approval label Mar 25, 2026

zhengshengning merged commit 390b532 into PaddlePaddle:develop Mar 26, 2026
185 of 213 checks passed

zhengshengning added a commit to zhengshengning/Paddle that referenced this pull request Mar 26, 2026

[PAA+OPT] Layer normalization accuracy alignment and performance opti…

a225fe2

…mization (PaddlePaddle#78321)

zhengshengning mentioned this pull request Mar 26, 2026

[Cherry-pick] [PAA+OPT] Layer normalization accuracy alignment and performance optimization #78321 #78490

Merged

EmmonsCurse mentioned this pull request Mar 27, 2026

[CI] Align with Paddle layer_norm kernel update PaddlePaddle/FastDeploy#7056

Merged

5 tasks

onecatcn mentioned this pull request Mar 30, 2026

[CI] Enable XPU-only PR CI skip for non-XPU hardware jobs #78524

Open

sneaxiy pushed a commit that referenced this pull request Mar 31, 2026

[Cherry-pick] [PAA+OPT] Layer normalization accuracy alignment and pe…

19c158a

…rformance optimization #78321 (#78490) * [PAA+OPT] Layer normalization accuracy alignment and performance optimization * fix * fix2

zhengshengning mentioned this pull request Apr 1, 2026

[Revert][PAA+OPT] Layer normalization accuracy alignment and performance optimization #78321 #78556

Closed

DrRyanHuang mentioned this pull request Apr 9, 2026

[Performance] Optimize LayerNorm CUDA kernels for unaligned / non-power-of-2 shapes #78262

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PAA+OPT] Layer normalization accuracy alignment and performance optimization#78321

[PAA+OPT] Layer normalization accuracy alignment and performance optimization#78321
zhengshengning merged 13 commits intoPaddlePaddle:developfrom
zhengshengning:acc_layer_norm

zhengshengning commented Mar 16, 2026 •

edited

Loading

Uh oh!

paddle-bot Bot commented Mar 16, 2026

Uh oh!

codecov-commenter commented Mar 19, 2026

Uh oh!

wanghuancoder left a comment

Uh oh!

lugimzzz left a comment

Uh oh!

DrRyanHuang left a comment

Uh oh!

This comment was marked as outdated.

Uh oh!

DrRyanHuang Mar 23, 2026

Uh oh!

DrRyanHuang Mar 23, 2026

Uh oh!

wanghuancoder left a comment

Uh oh!

From00 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

	? PADDLE_GET_CONST(float, op_desc.GetAttr("epsilon"))
	? PADDLE_GET_CONST(double, op_desc.GetAttr("epsilon"))

Conversation

zhengshengning commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Category

PR Types

Description

背景与动机

主要改动

是否引起精度变化

Uh oh!

paddle-bot Bot commented Mar 16, 2026

Uh oh!

codecov-commenter commented Mar 19, 2026

Codecov Report

Uh oh!

wanghuancoder left a comment

Choose a reason for hiding this comment

Uh oh!

lugimzzz left a comment

Choose a reason for hiding this comment

Uh oh!

DrRyanHuang left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

DrRyanHuang Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

DrRyanHuang Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

wanghuancoder left a comment

Choose a reason for hiding this comment

Uh oh!

From00 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

zhengshengning commented Mar 16, 2026 •

edited

Loading