vllm.v1.hisparse.binding
¶
Worker-side wiring of HiSparse caches to attention layers.
Functions:
-
allocate_hisparse_kv_caches–Allocate the host pool separately from the shared device backing.
-
bind_hisparse_kv_caches–Bind existing cache storage and block tables; return the bound handles.
-
fit_mamba_blocks_to_attention–Shrink align-mode Mamba blocks that would grow the scheduler block.
-
get_kv_sharing_readers–Cross-layer KV-sharing layers by target: they own no cache, so HiSparse
-
init_hisparse_kv_cache–Allocate and bind HiSparse caches within the caller's allocation context.
-
rebind_hisparse_device_caches–Rebind the resident and hot views over resized raw device storage.
-
resolve_hisparse_specs–Resolve a common kernel block size for HiSparse MLA specs, and add the
-
unbind_hisparse_caches–Detach HiSparse cache bindings without unregistering the host pool.
_largest_exact_kernel_block_size(block_size, backends, specs)
¶
Largest exact kernel page all backends accept, as a multiple of block_size.
Source code in vllm/v1/hisparse/binding.py
_tensor_placements(kv_cache_config)
¶
Each cache's tensor placement and byte offset.
Source code in vllm/v1/hisparse/binding.py
allocate_hisparse_kv_caches(kv_cache_config, device, layout, kernel_block_sizes, host_pool, allocate=None)
¶
Allocate the host pool separately from the shared device backing.
Source code in vllm/v1/hisparse/binding.py
bind_hisparse_kv_caches(*, forward_context, kv_cache_config, kv_caches, block_tables, host_pool)
¶
Bind existing cache storage and block tables; return the bound handles.
Source code in vllm/v1/hisparse/binding.py
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 | |
fit_mamba_blocks_to_attention(vllm_config, layer_configs, layers)
¶
Shrink align-mode Mamba blocks that would grow the scheduler block.
Mamba blocks start at cache_config.block_size, which resized attention
specs may no longer use. Rebuilds the Mamba specs at the largest size up to
it that divides the attention blocks' lcm.
Source code in vllm/v1/hisparse/binding.py
get_kv_sharing_readers(forward_context)
¶
Cross-layer KV-sharing layers by target: they own no cache, so HiSparse binds them to their target's resident, hot and host buffers.
Source code in vllm/v1/hisparse/binding.py
init_hisparse_kv_cache(kv_cache_config, device, kernel_block_sizes, vllm_config, forward_context, block_tables, allocate=None)
¶
Allocate and bind HiSparse caches within the caller's allocation context.
Source code in vllm/v1/hisparse/binding.py
rebind_hisparse_device_caches(*, forward_context, kv_cache_config, kv_caches)
¶
Rebind the resident and hot views over resized raw device storage.
kv_caches holds each resident and hot cache's raw storage, as allocation
returns it; as in bind_hisparse_kv_caches, resident entries become their
views and hot entries are dropped.
Source code in vllm/v1/hisparse/binding.py
resolve_hisparse_specs(vllm_config, layer_configs, attn_layers)
¶
Resolve a common kernel block size for HiSparse MLA specs, and add the resident/hot specs HiSparse derives from them, then fit align-mode Mamba blocks to the resized attention blocks.
Source code in vllm/v1/hisparse/binding.py
unbind_hisparse_caches(forward_context)
¶
Detach HiSparse cache bindings without unregistering the host pool.