vllm.models.minimax_m3.nvidia.specialized_decode.kernels.sparse_decode_kernel
¶
CuTe DSL cluster sparse decode (FP8 e4m3 KV, 16 selected pages of 128 tokens, MXFP8 output).
fp16-P online-softmax cluster decode. Every floating-point operation is spelled as an explicit PTX instruction
(fma.rn.f32, mul.f32 / add.f32 / sub.f32, ex2.approx.f32, rcp.rn.f32, div.rn.f32), so the
outputs do not depend on compiler contraction.
One cluster of CL CTAs (CL = 16 or 8) per (token row, KV head): CTA rank runs the selected pages
rank, rank + CL, ... (PPC = 16 / CL pages) with an online softmax in base 2 (S = q.k in fp32 from exact
fp8 -> fp16 operands on mma.sync m16n8k16, P rounded to fp16 before P @ V), then pushes its normalized partial of
Q head row and that head's log-sum-exp to CTA row / HPC over DSMEM (st.async completing an mbarrier);
CTA r merges HPC = HQ / CL heads (at least 1) by their LSE, writes the bf16 output and its MXFP8 quantization
(E4M3 values, UE8M0 scales in the 128x4 swizzled layout the o_proj GEMV reads plus a row-major copy).
HQ = Q heads per KV head: 16 (TP4) or 8 (TP8). The MMA keeps 16 rows; at 8 heads rows 8..15 of q are zero and their partials are dropped, and at CL 16 only CTAs 0..7 merge (one head each); the others finish after their push.
PDL: the CTA arrives on the cluster barrier, then waits on the grid dependency (griddepcontrol.wait) and at once
lets its dependents launch. prewait (PRE builds): bit 0 reads the lens before the wait (they are written at the
step start), bit 1 also q (written before the previous kernel's own grid wait returned); only the page tables are
the previous kernel's output. PRE = False reads every input after the wait.
Kernel arguments are raw device addresses (int64) and byte strides, so one compiled kernel serves every tensor of its (CL, HQ, PRE, min blocks, PDL) build.
Functions:
-
recv_bytes–Mbarrier transaction bytes a merging CTA expects: HPC * CL partials of 128 fp32 plus their LSE.
-
smem_alloc–Shared memory a CTA allocates: SMEM_BYTES, or (cap > 0) padded so that at most
capCTAs fit per SM (the
_Ctx
¶
Per-thread values the pieces share (a namespace, filled by the kernel).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_bf16_hi(x, *, loc=None, ip=None)
¶
bf16(x) (RN) in the high half of a word (= the fp32 bits of the rounded value).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_cvt16(words)
¶
16 E4M3 bytes (4 words) -> 16 fp16 (8 words), the CUDA cvt16 order.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_cvt_e4m3x2(w, hi, *, loc=None, ip=None)
¶
The fp16x2 of the E4M3 pair in the low (or high) 16 bits of w (exact).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_e4m3_lo(x, *, loc=None, ip=None)
¶
E4M3 (RN, satfinite) of x in the low byte (cvt.rn.satfinite.e4m3x2.f32 q2, 0, x).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_f1(op, a, *, loc=None, ip=None)
¶
Unary fp32 op (ex2.approx.f32, lg2.approx.f32, rcp.rn.f32, rcp.approx.ftz.f32, abs.f32).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_f16x2(lo, hi, *, loc=None, ip=None)
¶
__floats2half2_rn(lo, hi) as a word.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_f2(op, a, b, *, loc=None, ip=None)
¶
Binary fp32 op as an explicit PTX instruction (mul.f32, add.f32, sub.f32, max.f32, div.rn.f32).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_fetch(c, slot)
¶
Register prefetch of one selected page into c.kvr (K words 0..31, V words 32..63; thread covers token rows i * 16 + tid / 8, bytes (tid % 8) * 16) and its token count into c.ntr (0 when the slot is past the selection).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_ld_b32_if(addr, pred, other, *, loc=None, ip=None)
¶
pred ? *(int *)addr : other (no load when not pred).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_ld_v4_if(addr, pred, nc, *, loc=None, ip=None)
¶
16 bytes at addr (ld.global[.nc].v4) when pred, else zeros.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_lds_cvt(addr, *, loc=None, ip=None)
¶
The fp16x2 of the E4M3 pair at a shared address (2-byte load).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_merge(c)
¶
Merge the HPC heads of this CTA by their LSE; bf16 output + MXFP8 (values, swizzled and row-major scales).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_mma(c, a, b0, b1, *, loc=None, ip=None)
¶
mma.sync m16n8k16 row.col f32 += f16 x f16.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_page(c, j, slot)
¶
One selected page: stage K / V, S = q k^T, online softmax update, O += P V (state in c.st).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 | |
_push(c, r)
¶
Normalize the partial of Q row g + 8 r and push it with its LSE (log2) to CTA row / HPC, slot (row % HPC, rank). Rows >= HQ (the MMA's padding at 8 heads) push nothing.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_row_reduce(c, op, dst)
¶
dst[h] = op over the 4 warps' red[w][h] (threads h < 16; the others read row 0 and store nothing).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
_st_b16_hi(addr, w, *, loc=None, ip=None)
¶
Store the high 16 bits of w.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/sparse_decode_kernel.py
recv_bytes(hq, cl)
¶
Mbarrier transaction bytes a merging CTA expects: HPC * CL partials of 128 fp32 plus their LSE.
smem_alloc(cap)
¶
Shared memory a CTA allocates: SMEM_BYTES, or (cap > 0) padded so that at most cap CTAs fit per SM (the
residency that balances two waves of clusters better).