Optimize hunyuan3d#1240
Conversation
Quantize non-MoE Hunyuan3D DiT linear weights through fp8-sgl while keeping MoE weights in BF16. Reject checkpoints that still contain FP8 MoE weights.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for FP8 configurations, fused QK RMS normalization using Triton, and FlashInfer MoE backend integration with autotuning for Hunyuan3D. The reviewer feedback highlights two important issues: a hardcoded absolute path in the new FP8 configuration file that should be made relative, and a missing guard in _build_flashinfer_weights to prevent potential AttributeError exceptions when weights are not yet loaded.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| "autotune": true, | ||
| "tune_max_num_tokens": 8192 | ||
| }, | ||
| "dit_original_ckpt": "/data/nvme0/wangyingrui/Hunyuan3D-2.1-fp8/hunyuan3d-dit-v2-1/model.fp8.safetensors", |
There was a problem hiding this comment.
The configuration contains a hardcoded absolute path to a specific user's directory (/data/nvme0/wangyingrui/...). This will cause a FileNotFoundError for any other user or environment trying to run the FP8 pipeline. Please replace this with a relative path or a configurable placeholder.
| "dit_original_ckpt": "/data/nvme0/wangyingrui/Hunyuan3D-2.1-fp8/hunyuan3d-dit-v2-1/model.fp8.safetensors", | |
| "dit_original_ckpt": "hunyuan3d-dit-v2-1/model.fp8.safetensors", |
| def _build_flashinfer_weights(self): | ||
| fc1_list, fc2_list = [], [] | ||
| fc1_biases, fc2_biases = [], [] | ||
| for expert_w in self.experts: | ||
| fc1_list.append(expert_w.fc1._get_actual_weight().t().contiguous()) | ||
| fc2_list.append(expert_w.fc2._get_actual_weight().t().contiguous()) | ||
| fc1_biases.append(expert_w.fc1._get_actual_bias()) | ||
| fc2_biases.append(expert_w.fc2._get_actual_bias()) | ||
| self._fi_fc1_weight = torch.stack(fc1_list, dim=0) | ||
| self._fi_fc2_weight = torch.stack(fc2_list, dim=0) | ||
| self._fi_fc1_bias = self._stack_optional_biases(fc1_biases) | ||
| self._fi_fc2_bias = self._stack_optional_biases(fc2_biases) |
There was a problem hiding this comment.
The _build_flashinfer_weights method does not guard against unloaded weights. If to_cuda is called before the weights are loaded (which is common during initialization or offloading), _get_actual_weight() will return None, leading to an AttributeError when calling .t(). Adding an early return guard when weights are None makes both load and to_cuda robust against this issue.
| def _build_flashinfer_weights(self): | |
| fc1_list, fc2_list = [], [] | |
| fc1_biases, fc2_biases = [], [] | |
| for expert_w in self.experts: | |
| fc1_list.append(expert_w.fc1._get_actual_weight().t().contiguous()) | |
| fc2_list.append(expert_w.fc2._get_actual_weight().t().contiguous()) | |
| fc1_biases.append(expert_w.fc1._get_actual_bias()) | |
| fc2_biases.append(expert_w.fc2._get_actual_bias()) | |
| self._fi_fc1_weight = torch.stack(fc1_list, dim=0) | |
| self._fi_fc2_weight = torch.stack(fc2_list, dim=0) | |
| self._fi_fc1_bias = self._stack_optional_biases(fc1_biases) | |
| self._fi_fc2_bias = self._stack_optional_biases(fc2_biases) | |
| def _build_flashinfer_weights(self): | |
| if self.experts[0].fc1._get_actual_weight() is None: | |
| return | |
| fc1_list, fc2_list = [], [] | |
| fc1_biases, fc2_biases = [], [] | |
| for expert_w in self.experts: | |
| fc1_list.append(expert_w.fc1._get_actual_weight().t().contiguous()) | |
| fc2_list.append(expert_w.fc2._get_actual_weight().t().contiguous()) | |
| fc1_biases.append(expert_w.fc1._get_actual_bias()) | |
| fc2_biases.append(expert_w.fc2._get_actual_bias()) | |
| self._fi_fc1_weight = torch.stack(fc1_list, dim=0) | |
| self._fi_fc2_weight = torch.stack(fc2_list, dim=0) | |
| self._fi_fc1_bias = self._stack_optional_biases(fc1_biases) | |
| self._fi_fc2_bias = self._stack_optional_biases(fc2_biases) |
1 participant
Improve Hunyuan3D inference performance with reduced transformer sync overhead, FlashInfer MoE, fused QK RMSNorm, FP8 DiT support, and fused self-attention QKV projection. MoE FP8 remains disabled to avoid accuracy risk.