vllm.models.deepseek_v41.nvidia.specialized_decode.ampere.indexer
¶
The DeepSeek-V4.1 sparse-attention indexer on SM80.
The indexer scores every compressed key of a request against each query
token and keeps the index_topk best for sparse MLA. On SM100 its queries
and keys are MXFP4 and DeepGEMM computes the scores; SM80 has neither the
FP4 conversion instructions nor a DeepGEMM build. This module keeps the stock
indexer layer, its MXFP4 key cache, and vLLM's top-k and candidate selection,
and brings its own versions of the pieces that need DeepGEMM:
forward_q, which RoPEs the indexer queries and quantizes them to MXFP4, and_produce_k, which writes the MXFP4 keys;- the indexer op
dsv41_ampere_sparse_attn_indexer: the stock op's SM80 subset (MXFP4 cache, no context parallelism, no fused MQA+top-k) on the SM80 logits kernels ofmqa_logits.py; - the index K cache layer, whose metadata builder (
indexer_metadata.py) plans no DeepGEMM schedule.
The op scores prefill tokens chunk by chunk over keys gathered from the cache (the ragged kernel), and decode tokens one row each straight from the cache pages (the paged kernel). Both write FP32 logits from column zero, which the two-level candidate selection and the per-row top-k then read.
A decode step's logits are as wide as max_model_len (CUDA graphs fix the
shapes) while its rows are far shorter, and vLLM's shared selection kernels
size their work by that width. With the default top-k backend (auto),
the decode tokens therefore select with this package's kernels, which read
only each row's own length: topk.py selects the candidate blocks and the
top-k, and the candidate consumers score their candidate keys only (at most
2048 * 8 per row, however long the context) and select among them. They
select the same sets as the shared kernels over masked dense logits (ties at
the threshold go to the lowest positions) and list them in ascending order.
An explicit backend keeps vLLM's shared candidate and top-k kernels.
Classes:
-
AmpereIndexer–The DeepSeek-V4.1 indexer with SM80 query quantization and logits.
-
AmpereIndexerCache–The index K cache, with the SM80 metadata builder.
-
AmpereSparseAttnIndexer–The stock indexer layer on the SM80 indexer op.
Functions:
-
dsv41_ampere_sparse_attn_indexer–The indexer top-k of every token of the step into
-
init_ampere_all_gather–Build the Lamport all-gather of the row-sharded decode top-k.
AmpereIndexer
¶
Bases: DeepseekV4Indexer
The DeepSeek-V4.1 indexer with SM80 query quantization and logits.
Methods:
-
forward_q–MXFP4 indexer queries: values
[T, H, 64], scales as one int32
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_produce_k(latent, positions, rotary_emb)
¶
MXFP4 index keys of the group-boundary latents into the key cache:
k_norm(wk(latent)), RoPE'd at the group's first position.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
forward_q(qr, qr_scale, indexer_weights, positions, rotary_emb)
¶
MXFP4 indexer queries: values [T, H, 64], scales as one int32
per (token, head), and the FP32 head weights.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
AmpereIndexerCache
¶
Bases: DeepseekV4IndexerCache
The index K cache, with the SM80 metadata builder.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
AmpereSparseAttnIndexer
¶
Bases: SparseAttnIndexer
The stock indexer layer on the SM80 indexer op.
Methods:
-
forward_cuda–Top-k indices of the step's tokens;
kwas already written.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
forward_cuda(hidden_states, q_quant, k, weights)
¶
Top-k indices of the step's tokens; k was already written.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_decode_row_shard(rows, num_requests, world_size, rank)
¶
Rows [start, end) that rank scores in a pure-decode step and
the rows of a full share, or None when the step stays replicated.
Each rank takes ceil(num_requests / world_size) whole requests in
order, so the last ranks may take fewer or none. The split reads only
host state that every rank shares.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_decode_rows_topk(decode, kv_cache, q_values, q_scales, weights, topk_tokens, max_model_len, max_seq_len, topk_out, candidate_blocks, candidate_block_size, candidate_write, topk_backend, start, end, num_requests)
¶
Top-k of decode rows [start, end), whole requests, into
topk_out.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 | |
_decode_topk(metadata, kv_cache, q_values, q_scales, weights, topk_tokens, max_model_len, topk_indices_buffer, candidate_blocks, candidate_block_size, candidate_write, topk_backend)
¶
Top-k of the decode tokens, one row per token straight from the cache pages.
In a pure-decode step of at least ROW_SHARD_MIN_REQUESTS requests,
each TP rank scores and selects only its share of the rows, whole
requests when the decodes are uniform (_decode_row_shard), and a TP
all-gather of the int32 top-k rows rebuilds topk_indices_buffer
(_gather_topk_shares). The candidate consumers read their source's
index K cache and so take its share: a rank's consumers read only the
candidate blocks its own source wrote, and the candidate blocks are not
exchanged.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 | |
_gather_topk_shares(topk_indices_buffer, topk_tokens, rows, share_rows, filled, score)
¶
Score this rank's share of the decode rows and gather every rank's
share, in rank order, into the first rows rows of
topk_indices_buffer.
score writes the share's first filled top-k rows into the tensor
it is given; the rest of the share is -1. With SM80's Lamport all-gather
(ampere/all_gather.py) the shares are scored into and gathered in
place in the buffer's first world * share_rows rows (past rows
only -1 padding lands); otherwise into a private share, gathered with
the TP group's custom all-gather and copied.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_prefill_topk(metadata, kv_cache, q_values, q_scales, weights, topk_tokens, topk_indices_buffer, candidate_blocks, candidate_block_size, candidate_write, topk_backend)
¶
Top-k of the prefill tokens, chunk by chunk over gathered keys.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 | |
_reserve_profile_memory(device, head_dim, max_model_len, total_seq_lens, topk_indices_buffer)
¶
Allocate, once, the op's peak transient memory for the memory profile: the key-gather workspace, the larger of a prefill chunk's FP32 logits and a decode step's (with its top-k workspace), and the TP row-shard exchange.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_scale_bytes(scales)
¶
Four UE8M0 bytes packed as one int32 per (token, head) back to uint8
[..., 4].
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
_select(logits, row_ends, candidate_blocks, candidate_block_size, candidate_write, topk_backend)
¶
Two-level selection (V4.1): the candidate source indexer publishes its rows' best key blocks; the later indexers mask their scores to them.
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
dsv41_ampere_sparse_attn_indexer(hidden_states, k_cache_prefix, kv_cache, q_values, q_scale, weights, topk_tokens, head_dim, max_model_len, total_seq_lens, topk_indices_buffer, dense_mha_metadata_layer_name, candidate_blocks=None, candidate_block_size=0, candidate_write=False, topk_backend='auto')
¶
The indexer top-k of every token of the step into
topk_indices_buffer (request-local compressed positions, -1 padded).
Source code in vllm/models/deepseek_v41/nvidia/specialized_decode/ampere/indexer.py
520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 | |
init_ampere_all_gather(vllm_config)
¶
Build the Lamport all-gather of the row-sharded decode top-k. Collective over the TP group; the model calls it where it builds the SM80 boundaries' all-reduce, whose conditions (TP 2, 4 or 8 with full peer access) it needs.