Skip to content

vllm.model_executor.layers.quantization.utils.gptq_utils

Functions:

is_moe_layer_gptq_quantized(prefix, quantized_layers)

Whether the routed experts at prefix are quantized in the checkpoint.

prefix is the absolute name of the fused expert layer (...mlp.experts), while modules_in_block_to_quantize names the individual expert projections (...mlp.experts.0.down_proj) or the fused expert module itself, using either absolute or block-relative names. Match on the experts portion of the name so that a layer whose experts are stored unquantized -- e.g. the MTP layer of a partially quantized checkpoint -- is not given a quantized method.

Configs that enumerate no expert module at all say nothing about the experts, so the layer is assumed to be quantized.

Source code in vllm/model_executor/layers/quantization/utils/gptq_utils.py
def is_moe_layer_gptq_quantized(prefix: str, quantized_layers: list[str]) -> bool:
    """Whether the routed experts at `prefix` are quantized in the checkpoint.

    `prefix` is the absolute name of the fused expert layer
    (`...mlp.experts`), while `modules_in_block_to_quantize` names the
    individual expert projections (`...mlp.experts.0.down_proj`) or the fused
    expert module itself, using either absolute or block-relative names. Match
    on the `experts` portion of the name so that a layer whose experts are
    stored unquantized -- e.g. the MTP layer of a partially quantized
    checkpoint -- is not given a quantized method.

    Configs that enumerate no expert module at all say nothing about the
    experts, so the layer is assumed to be quantized.
    """
    expert_parents = set()
    for layer in flatten_list(quantized_layers):
        head, sep, _ = layer.partition(".experts.")
        parent = f"{head}.experts" if sep else layer
        if parent.rsplit(".", 1)[-1] == "experts":
            expert_parents.add(parent)

    if not expert_parents:
        return True

    # The runtime prefix can also be the parent of the experts module, e.g.
    # DBRX's `...ffn` with checkpoint names `...ffn.experts.*`.
    return any(
        prefix == parent
        or prefix.endswith(f".{parent}")
        or parent.startswith(f"{prefix}.")
        for parent in expert_parents
    )