fix: raise on non-finite qparams during pack-quantized compression - #854
fix: raise on non-finite qparams during pack-quantized compression#854rishabhsinha17 wants to merge 1 commit into
Conversation
Uncalibrated modules keep their torch.empty initialized scales; compressing them serializes garbage silently. Validate weight_scale and any float weight_zero_point before packing and raise a ValueError naming the module and tensor with non-finite counts. Related: vllm-project#832 Signed-off-by: Rishabh Sinha <rsinha17@terpmail.umd.edu>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
|
1 participant
Summary
Hardening for #832.
initialize_qparamsregisters scales withtorch.emptyand nothing zero-fills them, so a module that is targeted for quantization but never written by calibration carries uninitialized memory inweight_scale. Compression currently quantizes and packs with those scales without complaint, and the result is hard to spot after the fact: measured on 0.18.0 (CPU), every NaN-scale element quantizes to 0 and packs to the constant word0x88888888, while whole-tensor stats ofweight_packedstill look well distributed. Details and isolation matrix in this comment.This makes pack-quantized compression fail loudly at compress time instead of serializing garbage.
Changes
PackedQuantizationCompressor.compressvalidatesweight_scale(andweight_zero_pointwhen floating point) before quantizing and raisesValueErrorwith the non-finite count. The check sits after the meta-device early return, so the distributed meta pass is unaffected.ModelCompressor.compress_modelkeeps module names when collecting modules and wraps per-moduleValueErrorwith the module path, so the error names the offending module.Validation
CPU, torch 2.13.0, Python 3.11:
Full
tests/test_compressors/failure set is identical to main on this machine (75 pre-existing environment-specific failures, all CUDA/MPS related, zero introduced).make qualitypasses.