Skip to content

vllm.models.minimax_m3.nvidia.specialized_decode.kernels.post.kernel

CuTe DSL kernels of the post family (the attention -> MoE boundary of a sparse layer).

Every arithmetic op is spelled as an explicit PTX instruction (ftz / rn / approx variants, no contraction), so the outputs do not depend on compiler contraction. Cross-rank protocols: the prepushed two-parity exchange with the 0x80000000 empty sentinel, tagged rx router partials, tagged route lines, PDL trigger points.

Kernels:

  • post_route_kernel (allreduce_norm stage): one cluster of CL CTAs per row. Row warps land the W ranks' bf16 partials from this rank's exchange (poll + re-arm), add the residual, Gemma RMSNorm via a cluster st.async exchange of the row's sum of squares (and the |normed| argmax candidate), then the mode-Q per-token NVFP4 tail (and optional MXFP8). One extra routing warp (crank 0) polls the o_proj router CTAs' tagged partial logits, scales them by 1/rms and publishes the MiniMax2 top-4 as tagged route lines (optionally the fp32 logits).
  • mxfp8_rows_kernel: MXFP8 (UE8M0, 128x4-swizzled scales) of bf16 rows.

All buffers are passed as raw 64-bit device addresses (the host checks shapes, dtypes and alignment).

Functions:

  • better –

    (h1, l1) > (h0, l0) lexicographically, uint32 -> Boolean.

  • bf16_words_to_f32 –

    4 words (8 bf16) -> 8 fp32 (low half first).

  • bf16r –

    __bfloat162float(__float2bfloat16_rn(a))

  • bf16x2 –

    __floats2bfloat162_rn(a, b) as a word (a in the low half).

  • ctz –

    __ffs(x) - 1 for x != 0.

  • e4m3x4 –

    Four fp32 to E4M3 (RN, satfinite), v0 in the low byte (two __nv_fp8x2_e4m3 halves).

  • fimm –

    An fp32 constant with exactly these bits (kept out of the optimizer's constant folding).

  • gemma_w –

    (1 + w) of the gamma pair q (fp32, ftz add).

  • hi_f –

    The high bf16 of a word as fp32.

  • ld_nc_v4 –

    Read-only 16 bytes (ld.global.nc), ordered where written.

  • ld_v4 –

    16 bytes (zero when pred == 0), ordered where written (the post's residual may alias its residual_out).

  • lo_f –

    The low bf16 of a word as fp32.

  • mbar_wait_cluster0 –

    Spin until phase 0 of the CTA's barrier completes (acquire, cluster scope).

  • mxfp8_8 –

    lb::mxfp8_8: 4 lanes per 32-column block, warp-synchronous (stores predicated on active).

  • route_pub –

    route_pub: top-4 of the hoisted logits, then lane l stores words 2 (l & 1), +1 of replica l >> 1.

  • rx_ld –

    rx_ld: the lane's two 16-byte loads (experts 4 lane .. +3) of every rx block (unconditional).

  • rx_poll_sum –

    rx_poll_sum: poll the lane's rx words (exact tag match, globaltimer deadline), then S_e in rank order.

  • rx_take –

    rx_take: accept blocks whose four words carry tag (returns the new pending mask).

  • tanh_engine –

    Tanh as an explicit instruction sequence, so the routing keys are reproducible bit for bit (cute.math.tanh

  • twiddle –

    Order-preserving uint32 key of an fp32 (as Int32 bits; compare with ugt).

  • ue8m0_rp –

    __nv_cvt_float_to_e8m0(x, __NV_SATFINITE, cudaRoundPosInf)

  • ue8m0_to_f32 –

    float(__nv_fp8_e8m0): 2^(c - 127); c = 0 -> 2^-127 (0x00400000), c = 255 -> NaN.

  • ugt –

    A > b as uint32 -> Int32 0 / 1.

  • x4_quant8 –

    Shared tail of the NVFP4 16-column block quantizers: codes of xn * inv, sf byte from even lanes (stores

_b(cond)

Boolean -> Int32 0/1 (a store predicate).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def _b(cond):
    """Boolean -> Int32 0/1 (a store predicate)."""
    return cutlass.select_(cond, Int32(1), Int32(0))

_cex(a, b)

Compare-exchange on (hi, lo, sg) triples: a gets the larger.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def _cex(a, b):
    """Compare-exchange on (hi, lo, sg) triples: a gets the larger."""
    s = better(b[0], b[1], a[0], a[1])
    x = tuple(cutlass.select_(s, b[i], a[i]) for i in range(3))
    y = tuple(cutlass.select_(s, a[i], b[i]) for i in range(3))
    return x, y

_poll_exchange(world)

The post's exchange poll: re-load every rank's 16 bytes at p0 + r * 12288 that still holds the empty sentinel in any word, until none does (bounded spin, then trap). Returns the 4 * world words.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def _poll_exchange(world: int):
    """The post's exchange poll: re-load every rank's 16 bytes at p0 + r * 12288 that still holds the empty sentinel
    in any word, until none does (bounded spin, then trap). Returns the 4 * world words."""
    n = 4 * world

    @dsl_user_op
    def poll(p0, *, loc=None, ip=None):
        lds, chk = [], []
        for r in range(world):
            o = ", ".join(f"${4 * r + i}" for i in range(4))
            lds.append(
                f"\tand.b32 m, pend, {1 << r};\n\tsetp.ne.u32 q, m, 0;\n\t"
                f"@q ld.volatile.global.v4.s32 {{{o}}}, [${n}+{r * (H // 2) * 4}];\n"
            )
            ws = [f"${4 * r + i}" for i in range(4)]
            chk.append(
                f"\tand.b32 m, pend, {1 << r};\n\tsetp.ne.u32 q, m, 0;\n\t"
                f"setp.ne.and.s32 q, {ws[0]}, {K_EMPTY - (1 << 32)}, q;\n\t"
                f"setp.ne.and.s32 q, {ws[1]}, {K_EMPTY - (1 << 32)}, q;\n\t"
                f"setp.ne.and.s32 q, {ws[2]}, {K_EMPTY - (1 << 32)}, q;\n\t"
                f"setp.ne.and.s32 q, {ws[3]}, {K_EMPTY - (1 << 32)}, q;\n\t"
                f"@q and.b32 pend, pend, {~(1 << r) & 0xFFFFFFFF};\n"
            )
        text = (
            "{\n\t.reg .pred q;\n\t.reg .b32 pend, m;\n\t.reg .b64 spins;\n\t"
            f"mov.b32 pend, {(1 << world) - 1};\n\tmov.u64 spins, 0;\n\t"
            + "".join(f"\tmov.b32 ${i}, 0;\n" for i in range(n))
            + "PXL:\n"
            + "".join(lds)
            + "".join(chk)
            + f"\tadd.u64 spins, spins, 1;\n\tsetp.gt.s64 q, spins, {SPIN_CAP};\n\t@q trap;\n\t"
            "setp.ne.u32 q, pend, 0;\n\t@q bra PXL;\n\t}"
        )
        out = _asm(
            _llvm.StructType.get_literal([_T.i32()] * n),
            [Int64(p0)],
            text,
            ",".join(["=r"] * n) + ",l",
            side=True,
            loc=loc,
            ip=ip,
        )
        return _unpack(out, _T.i32(), n, Int32, loc, ip)

    return poll

better(h1, l1, h0, l0)

(h1, l1) > (h0, l0) lexicographically, uint32 -> Boolean.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def better(h1, l1, h0, l0):
    """(h1, l1) > (h0, l0) lexicographically, uint32 -> Boolean."""
    return (
        ugt(h1, h0) | (cutlass.select_(h1 == h0, Int32(1), Int32(0)) & ugt(l1, l0))
    ) != 0

bf16_words_to_f32(w)

4 words (8 bf16) -> 8 fp32 (low half first).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def bf16_words_to_f32(w):
    """4 words (8 bf16) -> 8 fp32 (low half first)."""
    out = []
    for q in range(4):
        out += [lo_f(w[q]), hi_f(w[q])]
    return out

bf16r(a, *, loc=None, ip=None)

__bfloat162float(__float2bfloat16_rn(a))

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def bf16r(a, *, loc=None, ip=None):
    """__bfloat162float(__float2bfloat16_rn(a))"""
    return Float32(
        _asm(
            _T.f32(),
            [Float32(a)],
            "{\n\t.reg .b16 h, z;\n\tcvt.rn.bf16.f32 h, $1;\n\tmov.b16 z, 0;\n\tmov.b32 $0, {z, h};\n\t}",
            "=f,f",
            loc=loc,
            ip=ip,
        )
    )

bf16x2(a, b, *, loc=None, ip=None)

__floats2bfloat162_rn(a, b) as a word (a in the low half).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def bf16x2(a, b, *, loc=None, ip=None):
    """__floats2bfloat162_rn(a, b) as a word (a in the low half)."""
    return Int32(
        _asm(
            _T.i32(),
            [Float32(a), Float32(b)],
            "cvt.rn.bf16x2.f32 $0, $2, $1;",
            "=r,f,f",
            loc=loc,
            ip=ip,
        )
    )

ctz(x, *, loc=None, ip=None)

__ffs(x) - 1 for x != 0.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ctz(x, *, loc=None, ip=None):
    """__ffs(x) - 1 for x != 0."""
    return Int32(
        _asm(
            _T.i32(),
            [Int32(x)],
            "{\n\t.reg .b32 r;\n\tbrev.b32 r, $1;\n\tclz.b32 $0, r;\n\t}",
            "=r,r",
            loc=loc,
            ip=ip,
        )
    )

e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None)

Four fp32 to E4M3 (RN, satfinite), v0 in the low byte (two __nv_fp8x2_e4m3 halves).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def e4m3x4(v0, v1, v2, v3, *, loc=None, ip=None):
    """Four fp32 to E4M3 (RN, satfinite), v0 in the low byte (two __nv_fp8x2_e4m3 halves)."""
    return Int32(
        _asm(
            _T.i32(),
            [Float32(v) for v in (v0, v1, v2, v3)],
            "{\n\t.reg .b16 lo, hi;\n\tcvt.rn.satfinite.e4m3x2.f32 lo, $2, $1;\n\t"
            "cvt.rn.satfinite.e4m3x2.f32 hi, $4, $3;\n\tmov.b32 $0, {lo, hi};\n\t}",
            "=r,f,f,f,f",
            loc=loc,
            ip=ip,
        )
    )

fimm(bits, *, loc=None, ip=None)

An fp32 constant with exactly these bits (kept out of the optimizer's constant folding).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def fimm(bits: int, *, loc=None, ip=None):
    """An fp32 constant with exactly these bits (kept out of the optimizer's constant folding)."""
    return Float32(
        _asm(_T.f32(), [], f"mov.b32 $0, 0f{bits:08X};", "=f", loc=loc, ip=ip)
    )

gemma_w(gw, q)

(1 + w) of the gamma pair q (fp32, ftz add).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def gemma_w(gw, q):
    """(1 + w) of the gamma pair q (fp32, ftz add)."""
    one = fimm(0x3F800000)
    return add_ftz(one, lo_f(gw[q])), add_ftz(one, hi_f(gw[q]))

hi_f(w, *, loc=None, ip=None)

The high bf16 of a word as fp32.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def hi_f(w, *, loc=None, ip=None):
    """The high bf16 of a word as fp32."""
    return Float32(
        _asm(
            _T.f32(), [Int32(w)], "and.b32 $0, $1, 0xFFFF0000;", "=f,r", loc=loc, ip=ip
        )
    )

ld_nc_v4(addr, *, loc=None, ip=None)

Read-only 16 bytes (ld.global.nc), ordered where written.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ld_nc_v4(addr, *, loc=None, ip=None):
    """Read-only 16 bytes (ld.global.nc), ordered where written."""
    out = _asm(
        _llvm.StructType.get_literal([_T.i32()] * 4),
        [Int64(addr)],
        "ld.global.nc.v4.u32 {$0, $1, $2, $3}, [$4];",
        "=r,=r,=r,=r,l",
        side=True,
        loc=loc,
        ip=ip,
    )
    return _unpack(out, _T.i32(), 4, Int32, loc, ip)

ld_v4(addr, pred, *, loc=None, ip=None)

16 bytes (zero when pred == 0), ordered where written (the post's residual may alias its residual_out).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ld_v4(addr, pred, *, loc=None, ip=None):
    """16 bytes (zero when pred == 0), ordered where written (the post's residual may alias its residual_out)."""
    out = _asm(
        _llvm.StructType.get_literal([_T.i32()] * 4),
        [Int64(addr), Int32(pred)],
        "{\n\t.reg .pred p;\n\tmov.b32 $0, 0;\n\tmov.b32 $1, 0;\n\tmov.b32 $2, 0;\n\tmov.b32 $3, 0;\n\t"
        "setp.ne.u32 p, $5, 0;\n\t@p ld.global.v4.u32 {$0, $1, $2, $3}, [$4];\n\t}",
        "=r,=r,=r,=r,l,r",
        side=True,
        loc=loc,
        ip=ip,
    )
    return _unpack(out, _T.i32(), 4, Int32, loc, ip)

lo_f(w, *, loc=None, ip=None)

The low bf16 of a word as fp32.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def lo_f(w, *, loc=None, ip=None):
    """The low bf16 of a word as fp32."""
    return Float32(
        _asm(_T.f32(), [Int32(w)], "shl.b32 $0, $1, 16;", "=f,r", loc=loc, ip=ip)
    )

mbar_wait_cluster0(bar, *, loc=None, ip=None)

Spin until phase 0 of the CTA's barrier completes (acquire, cluster scope).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def mbar_wait_cluster0(bar, *, loc=None, ip=None):
    """Spin until phase 0 of the CTA's barrier completes (acquire, cluster scope)."""
    _asm(
        None,
        [Int32(bar)],
        "{\n\t.reg .pred q;\n\tMBW:\n\tmbarrier.try_wait.parity.acquire.cluster.shared::cta.b64 q, [$0], 0;\n\t"
        "@!q bra MBW;\n\t}",
        "r,~{memory}",
        side=True,
        loc=loc,
        ip=ip,
    )

mxfp8_8(s, active, t, c0, out_q, out_sf, tid)

lb::mxfp8_8: 4 lanes per 32-column block, warp-synchronous (stores predicated on active).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def mxfp8_8(s, active, t, c0, out_q, out_sf, tid):
    """lb::mxfp8_8: 4 lanes per 32-column block, warp-synchronous (stores predicated on active)."""
    amax = Float32(0.0)
    for e in range(8):
        amax = fmax(amax, fabs(s[e]))
    amax = cutlass.select_(active != 0, amax, Float32(0.0))
    amax = fmax(amax, shfl_bfly_f(amax, 1))
    amax = fmax(amax, shfl_bfly_f(amax, 2))
    code = ue8m0_rp(mul_rn(amax, rcp_ftz(fimm(0x43E00000))))  # 448
    osc = cutlass.select_(
        amax != Float32(0.0), rcp_ftz(ue8m0_to_f32(code)), Float32(0.0)
    )
    p0 = e4m3x4(
        mul_rn(s[0], osc), mul_rn(s[1], osc), mul_rn(s[2], osc), mul_rn(s[3], osc)
    )
    p1 = e4m3x4(
        mul_rn(s[4], osc), mul_rn(s[5], osc), mul_rn(s[6], osc), mul_rn(s[7], osc)
    )
    st_v2_if(out_q + Int64(t) * Int64(H) + Int64(c0), p0, p1, active)
    kb = c0 // 32
    off = (
        (t // 128) * (SF_K_TILES * 512)
        + (kb // 4) * 512
        + (t % 32) * 16
        + ((t % 128) // 32) * 4
        + kb % 4
    )
    st_u8_if(out_sf + Int64(off), code, active & _b((tid & 3) == 0))

route_pub(route, repl, rmaxt, t, lane, tag, inv, S, bb)

route_pub: top-4 of the hoisted logits, then lane l stores words 2 (l & 1), +1 of replica l >> 1.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def route_pub(route, repl, rmaxt, t, lane, tag, inv, S, bb):
    """route_pub: top-4 of the hoisted logits, then lane l stores words 2 (l & 1), +1 of replica l >> 1."""
    c = []
    for i in range(4):
        x = mul_rn(inv, S[i])
        kk = add_rn(
            fma_rn(
                fimm(0x3F000000),
                tanh_engine(mul_rn(fimm(0x3F000000), x)),
                fimm(0x3F000000),
            ),
            bb[i],
        )
        c.append((twiddle(kk), Int32(65535) - (4 * lane + i), sub_rn(kk, bb[i])))
    c[0], c[1] = _cex(c[0], c[1])
    c[2], c[3] = _cex(c[2], c[3])
    c[0], c[2] = _cex(c[0], c[2])
    c[1], c[3] = _cex(c[1], c[3])
    c[1], c[2] = _cex(c[1], c[2])
    ek, sk = [], []
    zero = (Int32(0), Int32(0), Float32(0.0))
    for k in range(TOPK):
        mh = redux_max(c[0][0])
        ml = redux_max(cutlass.select_(c[0][0] == mh, c[0][1], Int32(0)))
        ek.append(Int32(65535) - ml)
        sk.append(shfl_idx_f(c[0][2], ek[k] >> 2))
        if k < TOPK - 1:  # the last shift is dead
            sh = c[0][1] == ml
            nc = [
                tuple(cutlass.select_(sh, c[j + 1][i], c[j][i]) for i in range(3))
                for j in range(3)
            ]
            nc.append(tuple(cutlass.select_(sh, zero[i], c[3][i]) for i in range(3)))
            c = nc
    total = add_rn(add_rn(sk[0], sk[2]), add_rn(sk[1], sk[3]))
    den = add_rn(total, fimm(SUM_EPS_BITS))
    h = (lane & 1) != 0
    s0 = cutlass.select_(h, sk[2], sk[0])
    s1 = cutlass.select_(h, sk[3], sk[1])
    e0 = cutlass.select_(h, ek[2], ek[0])
    e1 = cutlass.select_(h, ek[3], ek[1])
    w0 = bf16r(div_rn(mul_rn(s0, fimm(ROUTE_SCALE_BITS)), den))
    w1 = bf16r(div_rn(mul_rn(s1, fimm(ROUTE_SCALE_BITS)), den))
    hi = Int64(tag) << 32
    d0 = hi | (Int64(e0) << 16) | ((Int64(fbits(w0)) >> 16) & Int64(0xFFFF))
    d1 = hi | (Int64(e1) << 16) | ((Int64(fbits(w1)) >> 16) & Int64(0xFFFF))
    r0 = lane >> 1
    hoff = Int64((lane & 1) * 2)
    st_relaxed_v2u64_if(
        route
        + ((Int64(r0) * Int64(rmaxt) + Int64(t)) * Int64(LINE_U64) + hoff) * Int64(8),
        d0,
        d1,
        _b(r0 < repl),
    )
    st_relaxed_v2u64_if(
        route
        + ((Int64(r0 + 16) * Int64(rmaxt) + Int64(t)) * Int64(LINE_U64) + hoff)
        * Int64(8),
        d0,
        d1,
        _b(r0 + 16 < repl),
    )

rx_ld(row, lane, nb, buf)

rx_ld: the lane's two 16-byte loads (experts 4 lane .. +3) of every rx block (unconditional).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def rx_ld(row, lane, nb, buf):
    """rx_ld: the lane's two 16-byte loads (experts 4 lane .. +3) of every rx block (unconditional)."""
    for i in range(nb):
        a = row + Int64(i * E * 8) + Int64(lane) * Int64(32)
        v0 = ld_volatile_v2u64(a)
        v1 = ld_volatile_v2u64(a + Int64(16))
        buf[4 * i + 0] = v0[0]
        buf[4 * i + 1] = v0[1]
        buf[4 * i + 2] = v1[0]
        buf[4 * i + 3] = v1[1]

rx_poll_sum(row, lane, tag, deadline, nb, rx2, S)

rx_poll_sum: poll the lane's rx words (exact tag match, globaltimer deadline), then S_e in rank order. Returns (late, stale).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@cute.jit
def rx_poll_sum(
    row: Int64,
    lane: Int32,
    tag: Int32,
    deadline: Int64,
    nb: cutlass.Constexpr[int],
    rx2: cutlass.Constexpr[bool],
    S,
):
    """rx_poll_sum: poll the lane's rx words (exact tag match, globaltimer deadline), then S_e in rank order.
    Returns (late, stale)."""
    pending = Int32((1 << nb) - 1)
    acc = cute.make_rmem_tensor(cute.make_layout((4 * nb,)), Float32)
    a = cute.make_rmem_tensor(cute.make_layout((4 * nb,)), Int64)
    b = cute.make_rmem_tensor(cute.make_layout((4 * nb,)), Int64)
    for i in cutlass.range_constexpr(4 * nb):
        acc[i] = Float32(0.0)
    late = Int32(0)
    stale = Int32(0)
    t0 = Int64(0)
    go = Int32(1)
    rx_ld(row, lane, nb, a)
    if cutlass.const_expr(rx2):
        ti = globaltimer()
        while globaltimer() - ti < Int64(HALF_NS):
            pass
        while go != 0:
            rx_ld(row, lane, nb, b)
            pending = rx_take(tag, nb, a, pending, acc)
            if pending == 0:
                go = Int32(0)
            else:
                now = globaltimer()
                if t0 == Int64(0):
                    t0 = now
                elif now - t0 > deadline:
                    late = Int32(1)
                    stale = rx_stale(nb, a, pending)
                    go = Int32(0)
                if go != 0:
                    rx_ld(row, lane, nb, a)
                    pending = rx_take(tag, nb, b, pending, acc)
                    if pending == 0:
                        go = Int32(0)
                    else:
                        now = globaltimer()
                        if now - t0 > deadline:
                            late = Int32(1)
                            stale = rx_stale(nb, b, pending)
                            go = Int32(0)
    else:
        while go != 0:
            pending = rx_take(tag, nb, a, pending, acc)
            if pending == 0:
                go = Int32(0)
            else:
                now = globaltimer()
                if t0 == Int64(0):
                    t0 = now
                elif now - t0 > deadline:
                    late = Int32(1)
                    stale = rx_stale(nb, a, pending)
                    go = Int32(0)
                if go != 0:
                    rx_ld(row, lane, nb, a)
    if (
        late != 0
    ):  # keep whatever the last read held (rx_finish's garbage-tolerant values)
        for i in cutlass.range_constexpr(nb):
            if ((pending >> i) & 1) != 0:
                for j in cutlass.range_constexpr(4):
                    acc[4 * i + j] = ibits(Int32(a[4 * i + j] & Int64(0xFFFFFFFF)))
    for e in cutlass.range_constexpr(4):
        x = acc[e]
        for i in cutlass.range_constexpr(1, nb):
            x = add_rn(x, acc[4 * i + e])
        S[e] = x
    return late, stale

rx_take(tag, nb, buf, pending, acc)

rx_take: accept blocks whose four words carry tag (returns the new pending mask).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def rx_take(tag, nb, buf, pending, acc):
    """rx_take: accept blocks whose four words carry tag (returns the new pending mask)."""
    tg = Int64(tag) & Int64(0xFFFFFFFF)
    for i in range(nb):
        ok = (pending >> i) & 1
        for j in range(4):
            ok = ok & cutlass.select_(
                ((buf[4 * i + j] >> 32) & Int64(0xFFFFFFFF)) == tg, Int32(1), Int32(0)
            )
        for j in range(4):
            acc[4 * i + j] = cutlass.select_(
                ok != 0,
                ibits(Int32(buf[4 * i + j] & Int64(0xFFFFFFFF))),
                acc[4 * i + j],
            )
        pending = cutlass.select_(ok != 0, pending & Int32(~(1 << i)), pending)
    return pending

tanh_engine(x, *, loc=None, ip=None)

Tanh as an explicit instruction sequence, so the routing keys are reproducible bit for bit (cute.math.tanh lowers to a sequence with different bits).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def tanh_engine(x, *, loc=None, ip=None):
    """Tanh as an explicit instruction sequence, so the routing keys are reproducible bit for bit (cute.math.tanh
    lowers to a sequence with different bits)."""
    return Float32(
        _asm(
            _T.f32(),
            [Float32(x)],
            "{\n\t.reg .f32 a, b, c, d, e, f, g, h, i, j, k, l, m, n, o;\n\t.reg .pred p1, p2;\n\t"
            "abs.f32 a, $1;\n\tmul.f32 b, a, 0f4038AA3B;\n\tex2.approx.ftz.f32 c, b;\n\t"
            "add.f32 d, c, 0f3F800000;\n\trcp.approx.ftz.f32 e, d;\n\t"
            "fma.rn.f32 f, e, 0fC0000000, 0f3F800000;\n\tsetp.ge.f32 p1, a, 0f41102CB4;\n\t"
            "selp.f32 g, 0f3F800000, f, p1;\n\tcopysign.f32 h, $1, g;\n\tmul.f32 i, $1, $1;\n\t"
            "fma.rn.f32 j, i, 0f3C80F082, 0fBD563CAE;\n\tfma.rn.f32 k, j, i, 0f3E085941;\n\t"
            "fma.rn.f32 l, k, i, 0fBEAAA9ED;\n\tfma.rn.f32 m, l, i, 0f00000000;\n\t"
            "fma.rn.f32 n, m, $1, $1;\n\tsetp.ge.f32 p2, a, 0f3F19999A;\n\tselp.f32 $0, h, n, p2;\n\t}",
            "=f,f",
            loc=loc,
            ip=ip,
        )
    )

twiddle(x, *, loc=None, ip=None)

Order-preserving uint32 key of an fp32 (as Int32 bits; compare with ugt).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def twiddle(x, *, loc=None, ip=None):
    """Order-preserving uint32 key of an fp32 (as Int32 bits; compare with ugt)."""
    return Int32(
        _asm(
            _T.i32(),
            [Float32(x)],
            "{\n\t.reg .b32 b, n;\n\t.reg .pred p;\n\tmov.b32 b, $1;\n\tsetp.lt.s32 p, b, 0;\n\t"
            "not.b32 n, b;\n\tor.b32 b, b, 0x80000000;\n\tselp.b32 $0, n, b, p;\n\t}",
            "=r,f",
            loc=loc,
            ip=ip,
        )
    )

ue8m0_rp(x, *, loc=None, ip=None)

__nv_cvt_float_to_e8m0(x, __NV_SATFINITE, cudaRoundPosInf)

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ue8m0_rp(x, *, loc=None, ip=None):
    """__nv_cvt_float_to_e8m0(x, __NV_SATFINITE, cudaRoundPosInf)"""
    return Int32(
        _asm(
            _T.i32(),
            [Float32(x)],
            "{\n\t.reg .b16 h;\n\t.reg .b32 w;\n\tcvt.rp.satfinite.ue8m0x2.f32 h, 0f00000000, $1;\n\t"
            "cvt.u32.u16 w, h;\n\tand.b32 $0, w, 255;\n\t}",
            "=r,f",
            loc=loc,
            ip=ip,
        )
    )

ue8m0_to_f32(c, *, loc=None, ip=None)

float(__nv_fp8_e8m0): 2^(c - 127); c = 0 -> 2^-127 (0x00400000), c = 255 -> NaN.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ue8m0_to_f32(c, *, loc=None, ip=None):
    """float(__nv_fp8_e8m0): 2^(c - 127); c = 0 -> 2^-127 (0x00400000), c = 255 -> NaN."""
    return Float32(
        _asm(
            _T.f32(),
            [Int32(c)],
            "{\n\t.reg .b32 b;\n\t.reg .pred z, n;\n\tshl.b32 b, $1, 23;\n\tsetp.eq.u32 z, $1, 0;\n\t"
            "setp.eq.u32 n, $1, 255;\n\tselp.b32 b, 0x00400000, b, z;\n\tselp.b32 $0, 0x7FFFFFFF, b, n;\n\t}",
            "=f,r",
            loc=loc,
            ip=ip,
        )
    )

ugt(a, b, *, loc=None, ip=None)

A > b as uint32 -> Int32 0 / 1.

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
@dsl_user_op
def ugt(a, b, *, loc=None, ip=None):
    """A > b as uint32 -> Int32 0 / 1."""
    return Int32(
        _asm(
            _T.i32(),
            [Int32(a), Int32(b)],
            "{\n\t.reg .pred p;\n\tsetp.gt.u32 p, $1, $2;\n\tselp.u32 $0, 1, 0, p;\n\t}",
            "=r,r,r",
            loc=loc,
            ip=ip,
        )
    )

x4_quant8(xn, active, t, c0, inv_of, xq, xsf, tid, sfb)

Shared tail of the NVFP4 16-column block quantizers: codes of xn * inv, sf byte from even lanes (stores predicated on active).

Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/post/kernel.py
def x4_quant8(xn, active, t, c0, inv_of, xq, xsf, tid, sfb):
    """Shared tail of the NVFP4 16-column block quantizers: codes of xn * inv, sf byte from even lanes (stores
    predicated on active)."""
    w = Int32(0)
    for z in range(4):
        w = w | (
            e2m1x2(mul_rn(xn[2 * z + 1], inv_of), mul_rn(xn[2 * z], inv_of)) << (8 * z)
        )
    st_b32_if(xq + Int64(t) * Int64(H // 2) + Int64(c0 // 2), w, active)
    st_u8_if(
        xsf + Int64(t) * Int64(H // 16) + Int64(c0 // 16),
        sfb,
        active & _b((tid & 1) == 0),
    )