vllm.models.minimax_m3.amd.glue.kernel
¶
MiniMax-M3.1 decode glue (TP4, gfx950, 17 .. 192 rows): one persistent FlyDSL launch from the unreduced o_proj partial to the MoE inputs, ahead of AITER's A16W4 gemm1 / gemm2.
Stages of the default build (policy; stage walk, BLOCKS x THREADS, S rows, G = S / 4
rows a TP owner):
- rs push (CTAs 0 .. 127, 48 columns a task): this rank's partial, rows of owner g -> rank g's RS inbox [this rank]. Transport "lamport" (default): plain bf16 16-byte stores (lane = (row, vector)) into the owner's sentinel inbox, the 3 peers only; lpmix 4: a lane's j-th store goes to peer (j + row // 4) % 3, so each destination gets 384-byte (3-line) chunks. "tagged": tagged triple pairs (3 bf16 + 16-bit tag in one 64-bit store); mixdst: each store instruction's four 16-lane groups go to four different ranks.
- rs reduce (CTAs 0 .. 127): this rank's G rows, the 4 ranks' partials summed in rank
order (f32) -> bf16 h (the own partial read straight from
partin lamport builds), pushed to every rank's AG slot [this rank]. lpag (lamport, rows >= LP_AG_MIN_ROWS): plain bf16 16-byte stores into the AG sentinel inboxes, this rank's own first (ownfirst); else tagged triples. Mode 2 (the residual sharded by owner rows): bf16(h + this rank's residual rows). - own rows (orouter: CTAs 128 .. 128 + G - 1, which issued no remote stores): this rank's G owner rows (reduced here, landing first): h + residual -> residual out, Gemma RMSNorm -> normed (gemm1 input), MXFP8 copy; xfirst: normed stored, drained and XRDY published before residual out and the MXFP8 copy.
- router (CTAs 128 .. 255: 8 tiles of 16 experts x 16 K parts) on the G owner rows only: the FP32 weight (loaded at kernel entry) as bf16 hi + mid + lo (exact; rbf16) x normed on 16x16x16 bf16 MFMAs -> partial logits.
- route (CTAs 128 .. 128 + G - 1, a CTA an owner row): parts summed in order, sigmoid,
- bias (prefetched) for selection, top-4 (ties to the lower expert), weights = sigmoid / sum x 2.0 (the routed scale) -> tagged (id, weight) pairs into every rank's peer heap (the only cross-rank routing exchange).
- peer rows (CTAs 0 .. 127, and 128 + G .. past 128 rows): the 3G gathered rows: residual out, normed, MXFP8.
- sort (CTA 255): every rank's top-k pairs (one poll batch) -> top-k ids / weights for all rows + AITER moe_sorting's outputs (block 16); fastsort: one-wave ds_bpermute scan, (expert, slot) padding threads.
- re-arm (lamport): every CTA, after a barrier at its end, writes the sentinel back over the inbox slots it consumed (seam CTAs: their task's RS segments; row CTAs: their row's AG segments).
- finish (afinish): the CTA whose device-counter add completes the launch's BLOCKS moves the epoch.
Every option is a GlueBuild field (None = policy(rows), the measured per-width choice,
MI355X TP4; options measured and left off: rowvec, agdefer, rnorm, prepoll outside 48 ..
64 rows; lamport: lpmix 1, lpprobe, lpack).
mode: 0 = load only (compile), 1 = full residual (every row on every rank), 2 = residual sharded by owner rows.
Lamport transport (specialized_decode/mono/lamport.py): a payload is its own flag. The
inboxes (peer buffer past the tagged halves, two parities by epoch & 1) hold the
sentinel 0xFFFF in every half-word until a push lands; a reader re-loads until none of
the half-words it uses is the sentinel, and the CTA re-arms them after a barrier at its
end. The host fills the inboxes with 0xFF at setup and at every reset (arm_peer).
Numerics are the tagged build's: the same rank-order sums, the same row-stage lane
layout (outputs bitwise equal to the tagged kernel's).
Sync, tags, the device-side epoch and fail-stop are the upstream M3 monokernel's (specialized_decode/mono/sync.py, deepseek_v41 mono/common): every wait (tagged pair, triple or Lamport slot) is bounded by WAIT_TICKS; a timeout sets the error record [1, rank, epoch, site] on every rank and every waiter that sees it gives up.
Functions:
-
ag_push_all–Pair i of every rank's AG region := bits (mixdst: lane group q's j-th store to
-
ag_push_peers–Pair i of the three peers' AG regions := bits (mixdst: rotated by lane group).
-
arm_peer–Host side, no launch in flight: every Lamport inbox half-word := the sentinel
-
bf16_bits–The 16 bits of bf16(v) (round to nearest even), in an Int64.
-
finish–Every CTA marks the launch done; CTA 0 waits for the marks, then moves the epoch
-
lamport_span–(byte offset, bytes) of the Lamport inboxes in the peer buffer: 0xFF at setup and
-
lp_count_acked–lpack: this CTA's pushes acked (vmcnt 0 in every wave), then one add to the epoch
-
lp_peer–Rank p's peer buffer base (p traced).
-
lp_rearm_end–Consumers re-arm at their CTA's end: after a CTA barrier every lane of the CTA
-
lp_rearm_row–AG inbox row t (owner t // G, row t % G) on this rank: its 128 task segments, 768
-
lp_rearm_rs–The RS slots seam task
taskconsumed on this rank: 3 peer segments x G rows -
lp_row_ag–lamport: row t's 4 triples a lane from this rank's AG inbox (owner t // G) ->
-
lp_row_of–(has a row, the row t) of the row stage CTA
bidran (run_glue's placement). -
lp_seg–Byte address of segment (this launch's parity, rank
who, seam task, row r) of -
lp_wait_acked–lpack: until every seam CTA of this rank counted its acked RS pushes (thread 0
-
max_pad–sorted_ids length, as aiter.fused_moe.moe_sorting allocates it.
-
mfma_bf16–16x16x16 bf16 MFMA on 64-bit packed operands (4 bf16 each, element 0 =
-
peer_row–Peer row q (0 .. 3G - 1) in owner order rank + 1, rank + 2, rank + 3.
-
policy–The options a width builds with when its GlueBuild leaves them None (measured,
-
prepoll–One lane a wave spins on one pair before the CTA's full poll (a hint only: the
-
resolve–keywith every None option set bypolicy. -
router_weights–The task's FP32 weights (a lane's float4 x 3 chunks), issued at kernel entry.
-
rs_done_wait–Every seam CTA of this launch has reduced: one add each, then wait for the
-
sort_scan_wave–Exclusive scan of the padded counts by wave 0 (two experts a lane, ds_bpermute
-
split3–A float4 -> three 64-bit words of 4 bf16 (hi, mid, lo) with w = hi + mid + lo
-
stage_route–Row t: top-4 of sigmoid + bias (ties to the lower expert); weights = selected
-
stage_router–Experts 16 tile .. over K part
partfor every row (16-row tiles): f32 MFMAs, -
stage_router_rn–stage_routeron the owner rows with its own normalization (rnorm): rstd from -
stage_row–Row t: its owner's gathered h + residual (f32) -> residual out (bf16); Gemma
-
stage_row_vec–stage_rowwith vector stores: the f32 sums go through LDS; each thread then -
stage_rs_push–Triples 16 task ..: every row of this rank's partial to its owner's RS slot [this
-
stage_rs_push_lp–This rank's partial, owner g's rows, the task's 48 columns -> rank g's RS inbox
-
stage_rs_reduce–Triples 16 task .. of this rank's G rows: the 4 ranks' triples in rank order
-
stage_rs_reduce_batched–stage_rs_reducewith every pass's poll in one batch (rows past G poll row G - 1 -
stage_rs_reduce_lp3–lamport_rs: this rank's G rows of the task's columns in the tagged reducer's
-
stage_rs_reduce_lpv–lamport: this rank's G rows of the task's columns, lane = (row, 16-byte vector):
-
stage_sort–Every row's top-k -> AITER moe_sorting's outputs (block 16): expert segments in
-
stage_zero–moe_buf [S, HIDDEN] bf16 := 0, slice
partof ZERO_CTAS (gemm2 adds into it -
stamp–s_memrealtime ->
tl[CTA][point](timeline builds);namea TL name or a raw -
tl_participants–The CTAs that do the work a stamp closes (the harness reduces min / median / max
-
triple_bits–Row t's bf16 at columns 3 triple .. of a [*, HIDDEN] bf16 buffer, in bits
-
triple_value–Value k (0 .. 2) of a triple pair's (low word, high word).
_by_rank(c, own, got)
¶
[own] + got[d - 1] (peer at position d after this rank) -> the same words per absolute source rank.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
_err(c)
¶
This launch's error word reader (the glue's peer buffer starts with the error record).
_ld16(base, dw)
¶
_pad(col)
¶
LDS index of a row column with one pad word every 32 (a thread's 32-column MXFP8 group: distinct banks).
ag_push_all(c, i, bits)
¶
Pair i of every rank's AG region := bits (mixdst: lane group q's j-th store to rank (rank + 1 + q + j) % 4; ownfirst: this rank's own slot first, then the three peers rotated by lane group).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
ag_push_peers(c, i, bits)
¶
Pair i of the three peers' AG regions := bits (mixdst: rotated by lane group).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
arm_peer(peer_bytes_tensor)
¶
Host side, no launch in flight: every Lamport inbox half-word := the sentinel (the tagged regions untouched).
bf16_bits(v)
¶
finish(c, epoch, tag)
¶
Every CTA marks the launch done; CTA 0 waits for the marks, then moves the epoch on. afinish: every CTA adds one to a device counter; the CTA whose add completes a multiple of BLOCKS (the launch's last) moves the epoch.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lamport_span()
¶
(byte offset, bytes) of the Lamport inboxes in the peer buffer: 0xFF at setup and at every reset.
lp_count_acked(c)
¶
lpack: this CTA's pushes acked (vmcnt 0 in every wave), then one add to the epoch block's RS_DONE_WORD; thread 0 keeps the launch's target (the next multiple of SEAM_TASKS) in LDS for lp_wait_acked.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_peer(c, p)
¶
lp_rearm_end(c, bid)
¶
Consumers re-arm at their CTA's end: after a CTA barrier every lane of the CTA has consumed its slots.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_rearm_row(c, t)
¶
AG inbox row t (owner t // G, row t % G) on this rank: its 128 task segments, 768 sentinel stores.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_rearm_rs(c, task)
¶
The RS slots seam task task consumed on this rank: 3 peer segments x G rows
(contiguous), sentinel stores.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_row_ag(c, t, trs)
¶
lamport: row t's 4 triples a lane from this rank's AG inbox (owner t // G) -> [[(lo, hi)]] as the tagged poll.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_row_of(c, bid)
¶
(has a row, the row t) of the row stage CTA bid ran (run_glue's placement).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_seg(c, base, region, who, task, r)
¶
Byte address of segment (this launch's parity, rank who, seam task, row r) of
inbox region in the peer buffer at base.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
lp_wait_acked(c)
¶
lpack: until every seam CTA of this rank counted its acked RS pushes (thread 0 polls, bounded; then a barrier).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
max_pad(s)
¶
sorted_ids length, as aiter.fused_moe.moe_sorting allocates it.
mfma_bf16(a64, b64, acc)
¶
16x16x16 bf16 MFMA on 64-bit packed operands (4 bf16 each, element 0 = lowest k).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
peer_row(c, q)
¶
Peer row q (0 .. 3G - 1) in owner order rank + 1, rank + 2, rank + 3.
policy(rows)
¶
The options a width builds with when its GlueBuild leaves them None (measured, MI355X TP4). Options off: rowvec (+0..0.8 us), agdefer (+1.5..2.5: its RS finishes 7 us earlier but the deferred all-gather then blocks the owner rows' local stores), rnorm (+1..2.5: its local hand-offs queue behind the all-gather's remote stores), ownfirst (unmeasured), prepoll outside PREPOLL_ROWS.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
prepoll(c, addr, idx, site, triples=True)
¶
One lane a wave spins on one pair before the CTA's full poll (a hint only: the full poll checks every tag).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
resolve(key)
¶
key with every None option set by policy.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
router_weights(c, task)
¶
The task's FP32 weights (a lane's float4 x 3 chunks), issued at kernel entry. rnorm: a lane group's 12 consecutive columns (4 triples: the own AG slot's granularity) instead of 3 x 4 columns 16 apart.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
rs_done_wait(c)
¶
Every seam CTA of this launch has reduced: one add each, then wait for the launch's multiple of SEAM_TASKS.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
sort_scan_wave(c, padded)
¶
Exclusive scan of the padded counts by wave 0 (two experts a lane, ds_bpermute
Hillis-Steele) -> off[0 .. E], num_valid. fill was zeroed at the start.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
split3(w4)
¶
A float4 -> three 64-bit words of 4 bf16 (hi, mid, lo) with w = hi + mid + lo exactly (truncations).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_route(c, t)
¶
Row t: top-4 of sigmoid + bias (ties to the lower expert); weights = selected sigmoids x (2.0 / their sum).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 | |
stage_router(c, task, wv)
¶
Experts 16 tile .. over K part part for every row (16-row tiles): f32 MFMAs,
waves summed in order -> LOGIT[row][expert][part]. One wait for every row's XRDY,
then every tile's X loads in flight together.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 | |
stage_router_rn(c, task, wv)
¶
stage_router on the owner rows with its own normalization (rnorm): rstd from
the seam CTAs' SSQ partials (a row's 128 in a fixed order), v = own AG slot (h) +
residual for the lane group's 12 columns, x = bf16(v * rstd * (1 + w)) as the norm
CTAs compute it (rstd's summation order differs: a rare 1-ulp x difference).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 | |
stage_row(c, t)
¶
Row t: its owner's gathered h + residual (f32) -> residual out (bf16); Gemma RMSNorm -> normed (bf16), XRDY.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 | |
stage_row_vec(c, t)
¶
stage_row with vector stores: the f32 sums go through LDS; each thread then
owns 12 consecutive columns (24 bytes: res_out and normed as 64-bit stores). Same
math and bits.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 | |
stage_rs_push(c, task)
¶
Triples 16 task ..: every row of this rank's partial to its owner's RS slot [this rank]; peers' rows first.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_rs_push_lp(c, task)
¶
This rank's partial, owner g's rows, the task's 48 columns -> rank g's RS inbox
[this rank][task] as plain bf16 (lane = (row, 16-byte vector); every load before
the first store); the 3 peers only: the owner reads its own partial from part.
mixdst: the lane's j-th store goes to peer (j + row) % 3 (after this rank), so one
store instruction feeds all three links.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_rs_reduce(c, task)
¶
Triples 16 task .. of this rank's G rows: the 4 ranks' triples in rank order (f32) -> bf16 h -> every rank's AG slot [this rank].
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_rs_reduce_batched(c, task)
¶
stage_rs_reduce with every pass's poll in one batch (rows past G poll row G - 1
again).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_rs_reduce_lp3(c, task)
¶
lamport_rs: this rank's G rows of the task's columns in the tagged reducer's layout (lane = (row, triple); every pass's poll in one batch): each peer's segment read at the lane's triple, plus this rank's own partial (cached), summed in rank order (f32) -> _rs_sum -> tagged triples to every rank's AG slot, as the tagged build.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_rs_reduce_lpv(c, task)
¶
lamport: this rank's G rows of the task's columns, lane = (row, 16-byte vector):
the 3 peers' slots + this rank's own partial (cached), summed in rank order (f32) ->
bf16 h (mode 2: bf16(h + this rank's residual rows)) -> every rank's AG inbox
[this rank][task] as plain bf16 (ownfirst: this rank's own inbox first; mixdst:
the peers rotated by row).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stage_sort(c)
¶
Every row's top-k -> AITER moe_sorting's outputs (block 16): expert segments in ascending expert id, a pick's place in its segment by (row, slot), each segment padded to 16 with MOCK = (TOPK << 24) | S at weight 0, an expert id a 16-row block, num_valid = [padded rows, S]. Experts without picks get no block.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 1743 1744 1745 1746 1747 1748 1749 1750 1751 1752 1753 1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 1770 1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 1783 1784 1785 1786 1787 1788 1789 1790 1791 1792 1793 1794 1795 1796 1797 1798 1799 1800 1801 1802 1803 1804 1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 | |
stage_zero(c, part)
¶
moe_buf [S, HIDDEN] bf16 := 0, slice part of ZERO_CTAS (gemm2 adds into it
atomically).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
stamp(c, name)
¶
s_memrealtime -> tl[CTA][point] (timeline builds); name a TL name or a raw
index (post_kernel.py).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
tl_participants(s, orouter=False, afinish=False)
¶
The CTAs that do the work a stamp closes (the harness reduces min / median / max over them).
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
triple_bits(buf, t, triple)
¶
Row t's bf16 at columns 3 triple .. of a [*, HIDDEN] bf16 buffer, in bits 0 .. 47.
Source code in vllm/models/minimax_m3/amd/glue/kernel.py
triple_value(lo, hi, k)
¶
Value k (0 .. 2) of a triple pair's (low word, high word).