vllm.model_executor.layers.quantization.utils.lut_b_utils
¶
LUT-B weight format: 3-bit indices into per-tile E4M3 codebooks.
Rubin's tcgen05.mma ... decompress::lut::b instructions decode each weight
from a three-bit index into an eight-entry E4M3 codebook shared by eight
output channels. This module implements a calibration-free format matching
that contract. Each 8 (N) x 64 (K) tile of a [N, K] weight stores
192 bytes of packed three-bit indices (row-major over the tile) and an
eight-entry E4M3 codebook. Separately, every output channel carries one UE8M0
scale per 32 K ([N, K / 32]), which is exactly the hardware's block32 B
scale, so the 16 scales of a tile may differ. They keep small-magnitude rows
and blocks out of the E4M3 subnormal range. In total the format uses 3.375
bits per weight.
Functions:
-
dequantize_lut_b–Reconstruct the
[N, K]or[E, N, K]weight from LUT-B tiles. -
lut_b_to_sm107_tiles–Rearrange stacked LUT-B weights into the SM107 MoE kernel layout.
-
pack_lut_b_indices–Pack groups of eight 3-bit indices into three little-endian bytes.
-
quantize_lut_b–Quantize a
[N, K]or stacked[E, N, K]weight to LUT-B. -
unpack_lut_b_indices–Inverse of
pack_lut_b_indices.
_block_errors(blocks, exponent, tile_exponent, centers)
¶
Squared error of each K32 block, in units of the tile scale.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
_fit_tiles(values, weights, num_iterations)
¶
Fit one E4M3 codebook per row of values with weighted Lloyd iterations.
Three initializations are tried and the lowest-error fit per tile is kept.
Returns (indices, centers, loss), where loss is the weighted squared
error per tile.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
_normalize_centers(centers, preserve_zero)
¶
Snap to E4M3, keep an exact zero where needed, and sort.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
_refine_scales(blocks, exponent, tile_exponent, centers, num_rounds)
¶
Coordinate descent over block exponents and the tile codebook.
Each round moves every block exponent by -1, 0 or +1, whichever decodes that block best with the current codebook, then refits the codebook for the new exponents. Each step is kept only where it lowers the error.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
_rescale(blocks, exponent, tile_exponent)
¶
Per-tile values divided by their block scales, and Lloyd weights.
Each value is weighed by its scale squared, relative to the tile scale, so a codebook fit minimizes the error of the rescaled weights.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
_ue8m0_exponent(amax)
¶
Power of two that puts amax in (E4M3_MAX / 2, E4M3_MAX], as MXFP8.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
dequantize_lut_b(packed, codebooks, scales, out_dtype=torch.bfloat16)
¶
Reconstruct the [N, K] or [E, N, K] weight from LUT-B tiles.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
lut_b_to_sm107_tiles(packed, codebooks, scales)
¶
Rearrange stacked LUT-B weights into the SM107 MoE kernel layout.
Returns [E, N / 128, K / 128, 7168] uint8. Each tile holds:
- 6144 bytes of 3-bit indices, 48 bytes (K128) per output channel, in the no-swizzle K-major core-matrix layout (8 rows x 16 bytes),
- 512 bytes of UE8M0 B scales: byte
qof word[lane][col]is the scale of channelcol * 32 + lanefor K32 blockq, - 512 bytes of codebooks: lane
gholds the two eight-entry books (one per K64 half) of channel groupg; lanes 16-31 are zero padding.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
pack_lut_b_indices(indices)
¶
Pack groups of eight 3-bit indices into three little-endian bytes.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
quantize_lut_b(weight, *, num_iterations=LUT_B_LLOYD_ITERATIONS, num_scale_rounds=LUT_B_SCALE_ROUNDS, max_tiles_per_chunk=LUT_B_MAX_TILES_PER_CHUNK)
¶
Quantize a [N, K] or stacked [E, N, K] weight to LUT-B.
Returns (packed, codebooks, scales) with shapes
[E, N/8, K/64, 192] (uint8), [E, N/8, K/64, 8] (float8_e4m3fn)
and [E, N, K/32] (uint8 UE8M0). The expert dim is dropped for 2D
inputs.
Each tile is fitted twice: once with its 16 block scales tied to one
exponent for the whole tile, and once with each block's exponent taken
from its own amax. Starting from the better fit, num_scale_rounds rounds
of coordinate descent then adjust the block exponents and the codebook
together. Every step only lowers the reconstruction error, so the result
is never worse than a single scale per tile.
Source code in vllm/model_executor/layers/quantization/utils/lut_b_utils.py
218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 | |
unpack_lut_b_indices(packed)
¶
Inverse of pack_lut_b_indices.