Skip to content

vllm.models.muse_spark.amd.mono.kernels.draft

DSpark draft: the 5 draft layers and the final norm of a draft step as one persistent launch.

One rank of a TP8 group, 256 CTAs x 512 threads, S = 8 / 16 rows: 1 / 2 requests of 7 query rows (vLLM's rows 7 r + j; rows 7 NR .. S - 1 are padding: computed, never stored to vLLM's tensors). A draft layer is K3's attention block (k3.py) followed by a dense SwiGLU MLP:

IN     x = embeds -> the fp32 stream (smid), the input of the first boundary
BND    (FFN boundary of layer l - 1; the embeddings for l = 0) the owner of column block b
       (rank b // 32) sums the W down_proj partials in rank order (bf16, the all-reduce output),
       post-FFN norm (weight first, eps 1e-8), gated residual, the pre-QKV norm of layer l
       (eps 1e-5) of the bf16-rounded stream; the reduced rows and the norm partials all-gathered
QKV    layer l's qkv + o-gate GEMV (K in eighths, as K3); the owners finish whole heads: qk-norm,
       RoPE, q, the bf16 K / V into the layer's LBHNC draft KV cache (slot_mapping), the o-gate
ATT    after every CTA's QKV flag: the causal sliding-window attention over the draft KV
       (attn_frag.py, 7 query rows a request)
O      o-norm x sigmoid(gate), o_proj (this rank's K = 1024), partial to the block owner rank
AB     the attention boundary (as BND: post-attention norm weightless eps 1e-8, gated residual,
       pre-FFN norm) -> ffn_in
GU     gate_up (2 x 2048 rows a rank, K in eighths over 8-CTA groups; the partials go to the
       token owner CTA), bf16 gate / up, SwiGLU -> h (bf16, 2048 a rank)
DN     down_proj rows 32 b .. (K = 2048), partial to the block owner rank
after layer 4: BND with the final norm -> the draft hidden states (bf16, rows < 7 NR)

Every norm takes the bf16-rounded stream like the stock layer; its rstd comes from one token reduction round of the unrounded stream (k3.tok_rstd2's two-norm identity). The norm multipliers are fp32 (weight + gain center). Peer regions are double buffered by tag parity; the tags of a launch are epoch * 64 + 1 .. 6 (6 phases, so consecutive phases alternate parity across launches), and CTA 0 bumps the epoch at the end of the launch.

Functions:

  • build_vllm –

    The draft build vLLM launches for S rows (8 / 16: 1 / 2 requests of 7 rows), TP = 8.

  • peer_layout –

    Every rank's peer buffer (IPC-shared), [parity] first.

  • scratch_layout –

    This rank's scratch regions.

  • sfresh –

    An opaque s_mov of a wave-uniform i32 (dev.fresh for SGPRs).

build_vllm(S=8, W=8)

The draft build vLLM launches for S rows (8 / 16: 1 / 2 requests of 7 rows), TP = 8.

Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
def build_vllm(S=8, W=8):
    """The draft build vLLM launches for S rows (8 / 16: 1 / 2 requests of 7 rows), TP = 8."""
    assert W == 8 and S in (8, 16), (S, W)
    return Build(S=S)

peer_layout(S, world=W)

Every rank's peer buffer (IPC-shared), [parity] first.

Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
def peer_layout(S, world=W):
    """Every rank's peer buffer (IPC-shared), [parity] first."""
    assert world == W, world
    return _lay(
        [
            # o_proj / down_proj partials [par][src][t][pair] tagged
            ("op", 2 * W * S * (HID // 2) * 8),
            ("dp", 2 * W * S * (HID // 2) * 8),
            # the owners' reduced rows [par][t][pair] tagged (all-gather)
            ("orr", 2 * S * (HID // 2) * 8),
            ("yrr", 2 * S * (HID // 2) * 8),
            # the owners' norm partials [par][t][CTA][4] tagged (all-gather)
            ("ssa4p", 2 * 4 * S * BLOCKS * 8),
            ("ssg4p", 2 * 4 * S * BLOCKS * 8),
        ]
    )

scratch_layout(S, world=W)

This rank's scratch regions.

Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
def scratch_layout(S, world=W):
    """This rank's scratch regions."""
    assert world == W, world
    return _lay(
        [
            ("stream", S * HID * 4),  # the stream into a layer (fp32 plain)
            (
                "smid",
                S * HID * 4,
            ),  # the stream after the attention residual (fp32 plain)
            ("xin", S * HID * 2),  # ffn_in (bf16 plain), CTA b's 32 columns each
            ("xf", BLOCKS * 8),  # its flags
            ("xn", S * HID * 2),  # the layer's attention input (bf16 plain)
            ("xnf", BLOCKS * 8),
            # boundary norm rounds: [t][CTA][4] partials (unused: they come from the peers),
            # then (r1, r2)[t]
            ("ssa4", (4 * S * BLOCKS + 2 * S) * 8),
            ("ssg4", (4 * S * BLOCKS + 2 * S) * 8),
            # qkvg K-eighth partials [group][src][tile row][t]
            ("qx", 32 * 8 * NQT * 16 * S * 8),
            ("qf", BLOCKS * 8),  # QKV done flags
            # gate_up K-eighth partials [group][src][tile][row][t]
            ("gx", 32 * 8 * GUT * 16 * S * 8),
            ("h", S * FF * 2),  # SwiGLU output (bf16 plain)
            ("hf", BLOCKS * 8),
        ]
    )

sfresh(v)

An opaque s_mov of a wave-uniform i32 (dev.fresh for SGPRs).

Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
def sfresh(v):
    """An opaque s_mov of a wave-uniform i32 (dev.fresh for SGPRs)."""
    return fx.Int32(
        llvm.inline_asm(
            T.i32, [fx.Int32(v).ir_value()], "s_mov_b32 $0, $1", "=s,s",
            has_side_effects=True
        )
    )  # fmt: skip