vllm.models.muse_spark.amd.ops.fused_markov
¶
The Muse Spark DSpark Markov rounds with one launch a round (--monokernel).
DSparkSpeculator._sample_sequential runs many small kernels in each of the 7 draft rounds: the Markov embedding of the previous token, the bias GEMV over the 202048-token draft vocabulary (207 MB of markov_w2), the add to the base logits, Muse's output transform (multiplier and soft cap), the copy into the d2t scatter buffer, the Gumbel sample (its kernel, the argmax over its blocks and a gather) and the copy into draft_tokens.
Here a round is one launch over the vocabulary in blocks of BLOCK tokens. A
program gets the previous token of each row (the argmax over the previous
round's block maxima), computes its block's bias for all rows with one read
of markov_w2, and for each row adds the base logits, applies the transform,
stores the round's logits into the draft logits cache and takes the
Gumbel-max of its block with vLLM's own gumbel_noised_argmax. One small
launch then writes the last round's tokens. The noise depends only on the
request's seed, the round, the position and the token id, and both paths keep
the first of equal maxima, so the sampled token does not depend on the block
size.
The bias is an fp32 sum of 512 products rounded to bf16, as vLLM's GEMV output, but in another order. So a bias value can differ in its last bf16 bit. This changes only which tokens the draft proposes: the rejection sampler verifies every draft token against the target's distribution. The other steps are bitwise the same as vLLM's loop.
MuseSparkDSparkForCausalLM sets this module's sample_sequential as its
own with --monokernel on ROCm. tests/kernels/models/amd/
test_muse_spark_fused_markov.py compares it with vLLM's loop.
Functions:
-
sample_sequential–DSparkSpeculator._sample_sequential with one launch a Markov round.
_scratch(device, vocab, use_fp64)
¶
Two sets of block maxima and their token ids, one written by the current round and one read from the previous round. They are allocated once, so a CUDA graph keeps their addresses.
Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
_why_not(spec, num_reqs)
¶
Why this step takes vLLM's loop, or None when the fused launches compute the same sampling. Every check is the same on all ranks.
Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
sample_sequential(spec, num_reqs, head_hidden)
¶
DSparkSpeculator._sample_sequential with one launch a Markov round. Returns False, having done nothing, when this step takes vLLM's loop.
Source code in vllm/models/muse_spark/amd/ops/fused_markov.py
264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 | |