vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.gemv
¶
MXFP8 linear layers at decode batch sizes on SM80, in CuTe DSL.
DeepSeek-V4.1's dense projections are MXFP8: FP8 E4M3 weights W [N, K]
with one UE8M0 scale (2^(s - 127)) per 32 x 32 weight block. Before SM100
they run as BF16 cuBLAS GEMMs on weights dequantized once at load. A decode
step multiplies them by a handful of rows (M = 1..48: up to eight
requests of six tokens), where a GEMM reads its weights once and does almost
no arithmetic per byte, so its time is the weight read plus fixed costs.
This kernel computes y [M, N] = x [M, K] . dequant(W)^T (x and y
BF16) reading the FP8 weights, half the bytes of the BF16 copy.
Products. The kernel decodes each FP8 weight to BF16 in registers and
multiplies on BF16 tensor cores (mma.sync m16n8k16, FP32 accumulation):
weight rows are the MMA's M (16 per tile) and the rows of x its N (8 per
tile, up to six tiles), as the sparse MLA kernel does with keys and heads.
An E4M3 code s eeee mmm placed as the BF16 s | eeeemmm << 4 is its
value times 2^-120 (subnormal codes included, see codecs.py); one
fma.rn.bf16x2 by 2^(s - 7) (the block scale times 2^120) gives the
dequantized weight. That product is exact whenever the result is a normal
BF16, so every weight equals the one the load-time dequantization produces,
and the result differs from the cuBLAS path only in the FP32 summation
order. The combined factor must itself be a BF16, which holds for scale
bytes up to 134 (scales up to 2^7); pack_weight checks it.
Weight layout. pack_weight reorders the weights once, at load, into
512-byte blocks of 16 rows x 32 columns (one scale block's column range), in
which lane l of a warp finds, as 16 consecutive bytes, the 16 values its
two MMA A fragments need. A warp's load of one block is a single 16-byte
load per lane of 512 consecutive bytes. The K order inside a 32-column group
is permuted (both operands see the same permutation, which leaves the dot
product unchanged): lane (g, q) = (l / 4, l % 4) owns columns
8q .. 8q + 7 of rows g and g + 8, so its B fragments are the 16
consecutive bytes x[row, 8q .. 8q + 7], one 16-byte shared-memory read.
Within a word, bytes 0 and 2 are row g (two adjacent columns) and bytes
1 and 3 are row g + 8, so each word yields one BF16 pair of each row
with shifts and masks only. The scales become one word per 32 x 32 block
([N / 32, K / 32]): the factor 2^(s - 7) in both BF16 halves.
Work split. A CTA owns 16 warps_n weight rows and 32 warps_k groups
columns: warp (wn, wk) the 16 rows of tile wn and the groups
32-column groups at wk. All copies are cp.async to shared memory,
issued before any is used: first (one copy group) the CTA's columns of
x (rows past M zero-filled) and its scale factors, then each lane's
16 bytes of each of its warp's blocks, one copy group per block. A warp
then waits for its blocks one at a time (cp.async.wait_group), so it
decodes block i while the later ones are still in flight. (Loads into
registers would do the same with fewer instructions, but the assembler
sinks them next to their uses and keeps only a couple in flight.) After the
MMAs, the warps_k partial sums of each output meet in shared memory and
are added in warp order. When the columns are split over splits CTAs,
each CTA writes its FP32 partial sums to a workspace and takes an atomic
ticket; the last CTA of a row block adds the splits partial sums in
split order and writes BF16. The order of every FP32 addition is therefore
fixed: results are deterministic and do not depend on which CTA finishes
last. The ticket counters reset themselves (CUDA-graph safe). By default a
device's launches share one set, so they must be ordered on one stream at a
time; a caller whose launches can overlap on several streams gives each
weight its own counters.
default_config picks the split for a shape and device (its docstring
gives the model); mxfp8_gemv takes an explicit one too.
Classes:
-
GemvConfig–How a launch splits the work:
warps_nxwarps_kwarps per CTA, -
Mxfp8Gemv–y = x . dequant(W)^Tfor one weight shape andx_tilesx 8 rows
Functions:
-
copy_async–cp.asyncof 16 bytes to shared memory, of which the firstsize -
copy_async_weights–cp.asyncof 16 weight bytes to shared memory (L1 bypassed). -
copy_async_word–cp.asyncof one 4-byte word to shared memory. -
decode_pairs–The BF16 pairs of a packed weight word: bytes 0 and 2 (row
g) and -
default_config–The work split of a shape (module docstring).
-
load_partials–Four FP32 partial sums another CTA wrote, read from L2; the load stays
-
load_shared–16 bytes of shared memory (16-byte aligned) as four words.
-
load_shared_word–One word of shared memory.
-
mma_bf16–mma.syncm16n8k16 with BF16 operands and FP32 accumulators: the -
mxfp8_gemv–x [M, K] . dequant(W)^Tas BF16[M, N], forM <= 48. -
pack_weight–The kernel's layout of an MXFP8 weight (done once, at load).
-
store_pair–Store two words (8-byte aligned).
-
store_partials–Store four FP32 values (16-byte aligned).
GemvConfig
dataclass
¶
How a launch splits the work: warps_n x warps_k warps per CTA,
each with groups 32-column groups of one 16-row tile.
Methods:
-
splits–CTAs that share a row block's columns.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
Mxfp8Gemv
¶
y = x . dequant(W)^T for one weight shape and x_tiles x 8 rows
of x; see the module docstring.
Methods:
-
smem_bytes–Shared memory of one CTA (alignment padding included).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 | |
smem_bytes()
¶
Shared memory of one CTA (alignment padding included).
_asm(result, operands, text, constraints, side_effects, loc, ip)
¶
One inline PTX statement.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
_counters(device_index)
cached
¶
The ticket counters of a device (zero between launches).
_device_limits(device_index)
cached
¶
(SM count, opt-in shared memory per CTA in bytes) of a device.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
_kernel(device_index, n, k, x_tiles, config)
cached
¶
Mxfp8Gemv compiled for a device.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
_words(loaded, count, kind, wrap, loc, ip)
¶
The count members of an inline-assembly result struct.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
copy_async(shared, address, size, *, loc=None, ip=None)
¶
cp.async of 16 bytes to shared memory, of which the first size
(16 or 0) come from address and the rest are zeros.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
copy_async_weights(shared, address, *, loc=None, ip=None)
¶
cp.async of 16 weight bytes to shared memory (L1 bypassed).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
copy_async_word(shared, address, *, loc=None, ip=None)
¶
cp.async of one 4-byte word to shared memory.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
decode_pairs(word, scale)
¶
The BF16 pairs of a packed weight word: bytes 0 and 2 (row g) and
bytes 1 and 3 (row g + 8), each times the BF16x2 factor scale
(exact; module docstring).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
default_config(n, k, rows, device)
cached
¶
The work split of a shape (module docstring).
Candidates: CTAs of 1 to 16 warps (a power of two) that fit the
device's shared memory, at most 16 blocks per warp, between half and
four times as many CTAs as SMs (when any split allows it), and column
splits whose last CTA reads at most MAX_SPLIT_READ. It takes the
one whose busiest SM has the least modeled cost (the constants above,
fitted to sweeps of the DeepSeek-V4.1 shapes).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
load_partials(address, *, loc=None, ip=None)
¶
Four FP32 partial sums another CTA wrote, read from L2; the load stays after the ticket that orders it.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
load_shared(shared, *, loc=None, ip=None)
¶
16 bytes of shared memory (16-byte aligned) as four words.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
load_shared_word(shared, *, loc=None, ip=None)
¶
One word of shared memory.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
mma_bf16(c, a, b, *, loc=None, ip=None)
¶
mma.sync m16n8k16 with BF16 operands and FP32 accumulators: the
four accumulators of c plus the product of A (four words) and B (two
words).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
mxfp8_gemv(x, weight, scales, out=None, config=None, counters=None)
¶
x [M, K] . dequant(W)^T as BF16 [M, N], for M <= 48.
weight and scales come from pack_weight; x is BF16 with
16-byte aligned rows. config overrides default_config.
counters (int32, zero, at least N / 16) replaces the device's
shared ticket counters.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
pack_weight(weight, scales)
¶
The kernel's layout of an MXFP8 weight (done once, at load).
weight is [N, K] E4M3 (float8_e4m3fn or its bytes) and
scales the UE8M0 bytes [N / 32, K / 32], or [N, K / 32] with
each 32-row block's rows equal (the loader's row-expanded copy). Returns
the packed weights [N / 16, K / 32, 512] (uint8; module docstring)
and the scale factors [N / 32, K / 32] (int32): each block's
2^(s - 7) as BF16 in both halves of a word.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
store_pair(address, low, high, *, loc=None, ip=None)
¶
Store two words (8-byte aligned).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/gemv.py
store_partials(address, values, *, loc=None, ip=None)
¶
Store four FP32 values (16-byte aligned).