Skip to content

vllm.models.minimax_m3.nvidia.specialized_decode.kernels

Kernel registry of the MiniMax-M3 specialized decode (NVIDIA, SM100 family).

Maps each decode stage of specialized_decode/stages.py to the implementation that serves it and the envelope it is valid for: target arch, TP size and rows per rank (rows = c * (K + 1), c concurrent requests, K draft tokens). select returns None for any (stage, arch, tp, rows) outside every envelope; the caller then runs the stock path for that stage.

Nothing is imported or compiled at import time: KernelImpl.load() imports the implementing module lazily.

The kernels cover the sparse MoE layers 3-59; layers 0-2 run stock. The executor runs steps of up to MAX_ROWS = 64 rows (min(64, 8 * (K + 1))); each stage's envelope below is how far its kernel goes, and a step with more rows runs that stage stock (select returns None, fallback_stages names them for the build log):

==================== ======================================================= stage implementation ==================== ======================================================= front AR land + residual + RMSNorm + MXFP8 + QKV/index GEMV (tcgen05) + q/k norm + RoPE + FP8 KV / index-K insert index_score upstream minimax_m3_index_decode_score_cutedsl page_topk top-16 pages per (row, index head) sparse_decode 16-CTA cluster FP8 sparse decode with MXFP8 output o_proj MXFP8 o_proj GEMV + router CTAs, Lamport push allreduce_norm AR land + residual + norm + NVFP4/MXFP8 quant + top-4 moe fused NVFP4 routed + MXFP8 shared experts, push dense_mlp stock (no entry) ==================== =======================================================

Rows per stage (max_rows) and what serves the rows above it:

==================== ==== ================================================== stage rows above it ==================== ==== ================================================== front 16 the "wide" variant to 64 (the kernel's activation tile and its 20 boundary clusters are sized for 16 rows): stock AR + norm, MXFP8 quantize, the GEMV QKV / index projection (rows = MMA N), the stock q/k norm + RoPE + cache insert op index_score 64 (grid = requests; only heads x qlen <= 32 binds) page_topk 64 (grid = rows x heads) sparse_decode 64 (grid = rows x heads clusters; beyond one wave of clusters it runs in more than one wave) o_proj 8 the "wide" variant (wide.py) to 64: the same MXFP8 GEMV with the rows as the MMA's N (16 / 32 / 48 / 64), each weight tile read once; no router CTAs, no push (BF16 partial) allreduce_norm 8 stock (FlashInfer AR + Gemma norm, the stock router GEMM) moe 8 stock (at 32-64 rows x top-4 most experts are hit and the stock grouped GEMMs stream each weight once, at the HBM roofline; the fused engine's slots stop at 8) dense_mlp - stock at every row count ==================== ==== ==================================================

A stage's implementations are tried in order; the "wide" ones (variant) serve only the rows above the fused pair's (min_rows).

Modules:

  • allreduce_norm –

    allreduce_norm stage: the post-attention boundary of a sparse layer (TP4).

  • front –

    Stage front: the attention front of one sparse layer in one launch.

  • front_kernel –

    CuTe DSL device code of stage front (the host side is front.py).

  • geometry –

    Launch geometry of the specialized-decode kernels as a function of the SM

  • index_score –

    index_score stage: decode-time index page scores, served by upstream

  • moe –

    The moe stage of the MiniMax-M3 specialized decode: the fused NVFP4 routed +

  • o_proj –

    o_proj stage (TP4): MXFP8 o_proj GEMV + router CTAs, one CuTe DSL launch.

  • page_topk –

    page_topk stage: per (decode row, index head) the 16 pages the sparse

  • page_topk_kernel –

    page_topk device code (CuTe DSL): exact indexer top-16 page selector for the sparse-attention decode.

  • post –

    The post family (attention -> MoE boundary): CuTe DSL kernels (kernel.py) and

  • sparse_decode –

    sparse_decode stage: FP8-KV sparse decode over the 16 pages page_topk

  • sparse_decode_kernel –

    CuTe DSL cluster sparse decode (FP8 e4m3 KV, 16 selected pages of 128 tokens, MXFP8 output).

  • wide –

    The kernels of steps wider than the fused kernels' rows (9..64 rows).

Functions:

  • fallback_stages –

    Stages with an implementation that does not cover rows at

  • max_rows –

    The largest rows any variant implementation of stage runs.

  • select –

    The implementation serving stage for this envelope (of

fallback_stages(arch, tp, rows)

Stages with an implementation that does not cover rows at (arch, tp): they run stock for a step of that many rows.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/__init__.py
def fallback_stages(arch: str, tp: int, rows: int) -> tuple[str, ...]:
    """Stages with an implementation that does not cover `rows` at
    (arch, tp): they run stock for a step of that many rows."""
    return tuple(
        stage
        for stage in STAGES
        if any(arch in i.archs and tp in i.tps for i in REGISTRY.get(stage, ()))
        and select(stage, arch, tp, rows) is None
    )

max_rows(stage, variant='')

The largest rows any variant implementation of stage runs.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/__init__.py
def max_rows(stage: str, variant: str = "") -> int:
    """The largest rows any `variant` implementation of `stage` runs."""
    return max(
        (i.max_rows for i in REGISTRY.get(stage, ()) if i.variant == variant), default=0
    )

select(stage, arch, tp, rows, variant=None)

The implementation serving stage for this envelope (of variant, when given), or None (run the stock path).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/__init__.py
def select(
    stage: str, arch: str, tp: int, rows: int, variant: str | None = None
) -> KernelImpl | None:
    """The implementation serving `stage` for this envelope (of
    `variant`, when given), or None (run the stock path)."""
    if stage not in STAGES:
        raise KeyError(f"unknown stage {stage!r}; stages: {STAGES}")
    for impl in REGISTRY.get(stage, ()):
        if variant is not None and impl.variant != variant:
            continue
        if impl.supports(arch, tp, rows):
            return impl
    return None