vllm.models.minimax_m3.nvidia.specialized_decode.kernels.wide
¶
The kernels of steps wider than the fused kernels' rows (9..64 rows).
The fused o_proj / post / MoE engine kernels hold at most 8 rows (the engine's slot tiling, the router CTAs' register shape). Wider steps (c = 2..8 requests x 8 rows at K = 7) run these slots instead:
================== ===========================================================
stage wide implementation
================== ===========================================================
front (17..64) WideFront: after the stock input boundary (FlashInfer
AR + norm), one MXFP8 quantize launch and the same GEMV on
the QKV / index weight, then the stock q/k norm + RoPE +
FP8 KV / index-K insert op.
o_proj WideOProj: the same MXFP8 GEMV (tcgen05, swapped AB:
the weight is the M128 operand, the rows are the MMA's N =
16 / 32 / 48 / 64 columns), so each o_proj weight tile is
read once per step at any row count; it reads the kernel
sparse decode's MXFP8 output in place (no requantize).
BF16 partial out, no push.
allreduce_norm stock (FlashInfer's fused all-reduce + Gemma RMSNorm,
the stock router GEMM).
moe stock (above 8 rows the routed GEMMs stream nearly every
expert's weights once either way).
================== ===========================================================
Why not the push + post at these rows: the producer push writes 4-byte words to every peer, which is slower than FlashInfer's all-reduce + norm at 64 rows, and the MoE engine reads routes per slot. Why not the fused engine: at 32-64 rows x top-4 most of the 128 experts are hit and the routed GEMMs stream nearly every expert's weights once either way, which FlashInfer's grouped GEMMs already do near the HBM roofline. A separate router + NVFP4 launch with FlashInfer's routed entry point measured slower end to end at c = 4 / 8 than the stock router GEMM + monolithic MoE.
Classes:
-
WideFront–The QKV / index projection of a sparse layer at 17..64 rows: MXFP8 of
-
WideOProj–The o_proj MXFP8 GEMV at 9..64 rows on one layer's weight (no router
WideFront
¶
The QKV / index projection of a sparse layer at 17..64 rows: MXFP8 of
the normed BF16 rows (mxfp8_rows, trtllm mxfp8_quantize rounding)
and the GEMV on the layer's MXFP8 qkv_proj weight (rows =
q | k | v | index q | index k heads x 128).
Methods:
-
__call__–Qkv bf16 [rows, N] of the normed rows
hidden(bf16 [rows, 6144]). -
build–Load the quantize extension; compile the GEMV for
rows(once -
quantized–Qkv bf16 [rows, N] of rows already in MXFP8: e4m3
xq[rows,
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/wide.py
__call__(hidden)
¶
Qkv bf16 [rows, N] of the normed rows hidden (bf16 [rows, 6144]).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/wide.py
build(rows, compile_gemv=True)
¶
Load the quantize extension; compile the GEMV for rows (once
per process: the compile key does not depend on the weight).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/wide.py
quantized(xq, xsf)
¶
Qkv bf16 [rows, N] of rows already in MXFP8: e4m3 xq [rows,
6144] and UE8M0 xsf in the 128x4-swizzled layout (at least 128 x
192 bytes), as mxfp8_rows and the stock fused MoE finalize +
all-reduce + norm write them.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/wide.py
WideOProj
¶
The o_proj MXFP8 GEMV at 9..64 rows on one layer's weight (no router CTAs, no push): bf16 partial [rows, 6144] from the kernel sparse decode's MXFP8 output.
Methods:
-
build–Compile the MMA widths
rowsneed (outside capture).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/wide.py
build(rows)
¶
Compile the MMA widths rows need (outside capture).