Fix NVFP4 quantization for Qwen3.x MoE models (4 silent-failure bugs)#1323
Open
erictinkeredapps wants to merge 2 commits intoNVIDIA:mainfrom
Open
Fix NVFP4 quantization for Qwen3.x MoE models (4 silent-failure bugs)#1323erictinkeredapps wants to merge 2 commits intoNVIDIA:mainfrom
erictinkeredapps wants to merge 2 commits intoNVIDIA:mainfrom
Conversation
Four bugs prevent NVFP4 export from producing quantized weights for
Qwen3.5/3.6 MoE models (and potentially other fused MoE architectures).
All produce silent failures — no errors, just bfloat16 output identical
to input.
Bug 1: is_multimodal_model() crashes when config.architectures is None
- model_utils.py: add 'or []' fallback for NoneType iteration
Bug 3: get_quantization_format() doesn't recognize _QuantFusedExperts
- quant_utils.py: add check for plural ModuleList quantizers
(gate_up_proj_weight_quantizers, down_proj_weight_quantizers)
before the singular weight_quantizer loop
Bug 4: NVFP4 config wildcards don't match plural quantizer names
- config.py: _nvfp4_selective_quant_cfg() only generates patterns
for singular 'weight_quantizer', but _QuantFusedExperts creates
plural ModuleList quantizers. Add wildcard entries for both
gate_up_proj_weight_quantizers* and down_proj_weight_quantizers*
Bug 5: _process_quantized_modules elif order sends fused MoE to wrong path
- unified_export_hf.py: swap elif branches so hasattr check for
gate_up_proj_weight_quantizers comes before type-name checks.
Without this, QuantQwen3_5MoeExperts hits the singular-attribute
branch and crashes with AttributeError
Tested on: Qwen3.6-35B-A3B (MoE), NVIDIA DGX Spark (GB10),
modelopt 0.45.0 dev, transformers 5.5.4
Output: 20.5 GB NVFP4 (down from 66 GB bfloat16)
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughUpdates quantization export pipeline to support fused MoE expert modules: normalizes multimodal model detection, adds early quantizer-format detection for fused experts, reorders export logic to prioritize fused-expert handling, and extends NVFP4 selective quant config generation for plural weight-quantizer attributes. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@modelopt/torch/export/quant_utils.py`:
- Around line 673-685: The code currently only inspects quantizer_list[0] to
detect NVFP4, which misses cases where expert 0 is disabled; update the logic in
the detection block to iterate over quantizer_list and find the first quantizer
q where hasattr(q, "is_enabled") and q.is_enabled (or otherwise any enabled
quantizer), then read num_bits, block_sizes and compute scale_bits from that
enabled q and return QUANTIZATION_NVFP4 when matching (num_bits == (2, 1) and
scale_bits == (4, 3)); ensure the fallback to QUANTIZATION_NONE only happens
after checking all quantizers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f0da2041-19c7-4d5f-b7a8-ecfb8c942950
📒 Files selected for processing (4)
modelopt/torch/export/model_utils.pymodelopt/torch/export/quant_utils.pymodelopt/torch/export/unified_export_hf.pymodelopt/torch/quantization/config.py
Comment on lines
+673
to
+685
| if quantizer_list is not None and len(quantizer_list) > 0: | ||
| # Check the first quantizer in the list — all share the same config | ||
| q = quantizer_list[0] | ||
| if hasattr(q, "is_enabled") and q.is_enabled: | ||
| num_bits = getattr(q, "num_bits", None) | ||
| block_sizes = getattr(q, "block_sizes", None) | ||
| scale_bits = ( | ||
| block_sizes.get("scale_bits", (8, 0)) | ||
| if isinstance(block_sizes, dict) and "scale_bits" in block_sizes | ||
| else (8, 0) | ||
| ) | ||
| if num_bits == (2, 1) and scale_bits == (4, 3): | ||
| return QUANTIZATION_NVFP4 |
Contributor
There was a problem hiding this comment.
Avoid assuming expert quantizer at index 0 is representative.
At Line 675, only quantizer_list[0] is checked. If expert 0 is disabled and another expert is enabled, format detection can fall through to QUANTIZATION_NONE, causing a silent skip of quantized export for that fused module.
🔧 Suggested fix
- if quantizer_list is not None and len(quantizer_list) > 0:
- # Check the first quantizer in the list — all share the same config
- q = quantizer_list[0]
- if hasattr(q, "is_enabled") and q.is_enabled:
+ if quantizer_list is not None and len(quantizer_list) > 0:
+ # Find the first enabled quantizer in the list
+ q = next((item for item in quantizer_list if getattr(item, "is_enabled", False)), None)
+ if q is not None:
num_bits = getattr(q, "num_bits", None)
block_sizes = getattr(q, "block_sizes", None)
scale_bits = (
block_sizes.get("scale_bits", (8, 0))
if isinstance(block_sizes, dict) and "scale_bits" in block_sizes
else (8, 0)
)
if num_bits == (2, 1) and scale_bits == (4, 3):
return QUANTIZATION_NVFP4🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@modelopt/torch/export/quant_utils.py` around lines 673 - 685, The code
currently only inspects quantizer_list[0] to detect NVFP4, which misses cases
where expert 0 is disabled; update the logic in the detection block to iterate
over quantizer_list and find the first quantizer q where hasattr(q,
"is_enabled") and q.is_enabled (or otherwise any enabled quantizer), then read
num_bits, block_sizes and compute scale_bits from that enabled q and return
QUANTIZATION_NVFP4 when matching (num_bits == (2, 1) and scale_bits == (4, 3));
ensure the fallback to QUANTIZATION_NONE only happens after checking all
quantizers.
CodeRabbit review: expert 0 may be disabled when uncalibrated, so checking only quantizer_list[0] can miss the actual NVFP4 config. Now iterates to find the first enabled quantizer in the list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four bugs prevent NVFP4 quantization from producing quantized weights for Qwen3.5/3.6 MoE models (and likely other fused MoE architectures using
_QuantFusedExperts). All four produce silent failures — no errors, just bfloat16 output identical to the input model.Test Environment
Bug Details
Bug 1:
is_multimodal_model()crashes onNonearchitecturesFile:
modelopt/torch/export/model_utils.pyModels with
config.architectures = None(common for fine-tuned checkpoints) crash whenis_multimodal_model()iterates the list. One-line fix:or []fallback.Bug 2: (Usage issue, not a code bug — fixed in caller)
Bug 3:
get_quantization_format()does not recognize_QuantFusedExpertsFile:
modelopt/torch/export/quant_utils.pyThe function iterates
weight_attr_names(module)which returns singular attribute names._QuantFusedExpertsmodules use plural ModuleList quantizers (gate_up_proj_weight_quantizers.N), so the function returnsNoneand the module is treated as unquantized. Added a pre-check for plural ModuleList quantizers before the singular loop.Bug 4: NVFP4 config wildcards do not match plural quantizer names
File:
modelopt/torch/quantization/config.py_nvfp4_selective_quant_cfg()generates patterns like*mlp.experts*weight_quantizer(singular)._QuantFusedExpertscreates quantizers namedgate_up_proj_weight_quantizers.0(plural + index). Thefnmatchfails, quantizers never receive NVFP4 config, and 100% stay at disabled default. Added wildcard entries for both plural suffix patterns.Bug 5:
_process_quantized_moduleselif order sends fused MoE to wrong export pathFile:
modelopt/torch/export/unified_export_hf.pyTwo elif branches: one checks type name (
"Llama4TextExperts" in type().__name__), the other checkshasattr("gate_up_proj_weight_quantizers"). After_QuantFusedExpertswrapping,QuantQwen3_5MoeExpertsmatches the type-name branch, which calls_export_quantized_weight()looking for singular attributes →AttributeError. Swapped the elif order so the plural-attribute check runs first.Changes
modelopt/torch/export/model_utils.pyor []fallbackmodelopt/torch/export/quant_utils.pymodelopt/torch/export/unified_export_hf.pymodelopt/torch/quantization/config.pySummary by CodeRabbit
Bug Fixes
New Features