vllm.model_executor.layers.quantization.online.int4
¶
Online int4 weight-only quantization.
Int4PerGroup32AsymLMHeadMethod quantizes an fp16/bf16 lm_head at load time
(a quantization_config.targets pattern naming the lm_head, with
{"weight": "int4_per_group_32_asym"}). Decode reads the
whole lm_head every step, so on bandwidth-bound GPUs a full-precision head is
one of the largest per-token costs even when the checkpoint quantizes every
other linear layer.
Classes:
-
Int4PerGroup32AsymLMHeadMethod–Weight-only asymmetric uint4 lm_head, one scale and zero point per 32
Functions:
-
quantize_int4_per_group_asym–Asymmetric uint4 quantization of the rows of
weight[N, K]. -
quantize_int4_to_mp_layout–Quantize
weight[N, K] into theMPLinearKerneluint4 layout.
Int4PerGroup32AsymLMHeadMethod
¶
Bases: OnlineLMHeadMethodBase
Weight-only asymmetric uint4 lm_head, one scale and zero point per 32 input channels.
Source code in vllm/model_executor/layers/quantization/online/int4.py
quantize_int4_per_group_asym(weight, group_size)
¶
Asymmetric uint4 quantization of the rows of weight [N, K].
Returns the codes [N, K] uint8, the scales [N, K // group_size] in the
weight dtype and the zero points [N, K // group_size] uint8, such that
(codes - zero_point) * scale approximates weight. Each group's
range is widened to include 0, so the zero point always fits in uint4.
Source code in vllm/model_executor/layers/quantization/online/int4.py
quantize_int4_to_mp_layout(weight, group_size, chunk_rows=8192)
¶
Quantize weight [N, K] into the MPLinearKernel uint4 layout.
Returns the codes packed along K (int32 [N, K // 8]), the scales [N, K // group_size] and the zero points packed along N (int32 [N // 8, K // group_size]), the layout of compressed-tensors asymmetric W4A16 checkpoints. Rows are processed in chunks to bound the fp32 working set.