vllm.models.minimax_m3.nvidia.specialized_decode.kernels.geometry
¶
Launch geometry of the specialized-decode kernels as a function of the SM count (and, for cluster kernels, of the occupancy the CUDA runtime reports).
Pure Python (no torch, no CUDA), so every choice can be checked on CPU for any
SM count. The stages call these functions with the live
_arch.DeviceSpec.sm_count and the probed occupancy; nothing here is a
substitute for the build-time residency checks in _arch, which still fail
closed.
Which launch dimensions depend on the SM count:
================= ================================================== =========
stage grid SM-bound?
================= ================================================== =========
front 4 x (heads + 1) CTAs, all spinning residency
sparse_decode rows x Hkv clusters of 16 or 8 CTAs width, waves
o_proj (6144 / 128) x 2 + 16 CTAs (shape-derived) no
allreduce_norm rows clusters of cl CTAs, all spinning residency
moe one CTA per SM: routed = SMs - shared CTAs split
page_topk rows x column splits CTAs, no cross-CTA wait no
index_score upstream kernel, (batch, 256) CTAs, no waits no
================= ================================================== =========
Functions:
-
expected_distinct_experts–E[distinct experts] of
rowstokens choosingtopkofexperts -
moe_fc2_blocks_per_cta–FC2 32-row output blocks per routed CTA, ceil(6144 / 32 / routed); the
-
moe_n_shared–Shared-expert CTAs of the engine for
rowstokens onsm_count -
moe_n_shared_model–The valid shared-CTA count with the lowest modelled cost (ties: more
-
moe_routed_ctas–Routed CTAs of the engine: every SM not running a shared CTA.
-
moe_step_cost–Bytes-per-CTA model of one engine call: the slower of the routed CTAs
-
sparse_cluster_size–Cluster width of the sparse decode for
rowsxnum_kv_heads -
sparse_waves–Waves of
rowsxnum_kv_headsclusters of widthcl(inf if
expected_distinct_experts(rows, experts=_E, topk=_TOPK)
¶
E[distinct experts] of rows tokens choosing topk of experts
uniformly (the routed weight bytes a call streams scale with this).
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
moe_fc2_blocks_per_cta(routed_ctas)
¶
FC2 32-row output blocks per routed CTA, ceil(6144 / 32 / routed); the engine's shared-memory budget was sized for at most 2.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
moe_n_shared(rows, sm_count)
¶
Shared-expert CTAs of the engine for rows tokens on sm_count
(live) SMs: the model, unless a measured override exists for that count.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
moe_n_shared_model(rows, sm_count, routed_cost=MOE_ROUTED_COST)
¶
The valid shared-CTA count with the lowest modelled cost (ties: more
shared CTAs) that leaves at least MOE_MIN_ROUTED_CTAS routed CTAs.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
moe_routed_ctas(rows, sm_count, n_shared=None)
¶
Routed CTAs of the engine: every SM not running a shared CTA.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
moe_step_cost(rows, sm_count, n_shared, routed_cost=MOE_ROUTED_COST)
¶
Bytes-per-CTA model of one engine call: the slower of the routed CTAs (the call's distinct experts over SMs - shared CTAs) and the shared CTAs.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
sparse_cluster_size(rows, num_kv_heads, max_active_clusters=None, *, one_wave=True)
¶
Cluster width of the sparse decode for rows x num_kv_heads
clusters. The preferred width is 16 below SPARSE_WIDE_CLUSTERS
clusters, else 8. With max_active_clusters (width -> clusters the
device holds at once, cudaOccupancyMaxActiveClusters):
one_wave: the first width, from the preferred one down, whose grid is co-resident;ValueErrorif none is (fail closed);- otherwise the grid only has to be schedulable: the width with the
fewest waves, the wider one on a tie (more CTAs per row, shorter
per-CTA page loop);
ValueErrorif no width is schedulable.
Source code in vllm/models/minimax_m3/nvidia/specialized_decode/kernels/geometry.py
sparse_waves(rows, num_kv_heads, cl, max_active_clusters)
¶
Waves of rows x num_kv_heads clusters of width cl (inf if
that width is not schedulable).