Skip to content

vllm.model_executor.model_loader.weight_cache.weight_seed

Daemon-to-daemon weight shard seeding (movers, the seedable, the peer loader).

Modules:

  • core –

    Weight movers: copy whole storages so views keep their sharing and footprint.

  • loader –

    Fill a daemon's model with a source replica's post-processed weights.

  • mooncake –

    The Mooncake weight mover: pull storages through the TransferEngine.

  • peer_ipc –

    The peer_ipc weight mover: copy storages through CUDA IPC on one host.

  • seed –

    The weights seedable: a daemon both serves and fetches its shard.

  • utils –

    Helpers for weight shard seeding (manifest format, fingerprint, launcher logic).

Classes:

  • WeightMover –

    One mover that copies weight storages from a source daemon.

  • WeightShardSeed –

    This daemon's weight shard: served to and fetched from peer daemons.

Functions:

  • build_manifest –

    Describe tensors as views over their unique storages.

  • get_mover –

    The mover for a backend; device_index pins the peer_ipc device.

  • rebuild_tensors –

    Recreate every manifest tensor on the mirror's copies; bounds-checked first.

WeightMover

Bases: ABC

One mover that copies weight storages from a source daemon.

Methods:

  • copy_into –

    Copy every source storage into its local byte buffer.

  • fill –

    Copy the source's storages and rebuild its tensors on them.

  • prepare_seed –

    Publish how a mirror can read each storage, in manifest order.

  • release –

    Drop source-side state for the weights the daemon is releasing.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
class WeightMover(ABC):
    """One mover that copies weight storages from a source daemon."""

    name: SeedBackend

    @abstractmethod
    def prepare_seed(self, storages: list[torch.UntypedStorage]) -> dict[str, Any]:
        """Publish how a mirror can read each storage, in manifest order."""

    @abstractmethod
    def copy_into(
        self,
        storages: list[Mapping[str, Any]],
        seed: Mapping[str, Any],
        buffers: list[torch.Tensor],
    ) -> None:
        """Copy every source storage into its local byte buffer."""

    def fill(
        self,
        manifest: Mapping[str, Any],
        seed: Mapping[str, Any],
        device: torch.device,
    ) -> dict[str, torch.Tensor]:
        """Copy the source's storages and rebuild its tensors on them."""
        started = time.perf_counter()
        storages = manifest["storages"]
        buffers = [allocate_storage(storage, device) for storage in storages]
        self.copy_into(storages, seed, buffers)
        tensors = rebuild_tensors(manifest, buffers)
        total_gib = manifest_nbytes(manifest) / 1024**3
        elapsed = time.perf_counter() - started
        logger.info(
            "Mirrored %.2f GiB in %d storage(s) over %s in %.2fs (%.1f GiB/s)",
            total_gib,
            len(buffers),
            self.name,
            elapsed,
            total_gib / max(elapsed, 1e-9),
        )
        return tensors

    def release(self) -> None:
        """Drop source-side state for the weights the daemon is releasing."""
        return None

copy_into(storages, seed, buffers) abstractmethod

Copy every source storage into its local byte buffer.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
@abstractmethod
def copy_into(
    self,
    storages: list[Mapping[str, Any]],
    seed: Mapping[str, Any],
    buffers: list[torch.Tensor],
) -> None:
    """Copy every source storage into its local byte buffer."""

fill(manifest, seed, device)

Copy the source's storages and rebuild its tensors on them.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
def fill(
    self,
    manifest: Mapping[str, Any],
    seed: Mapping[str, Any],
    device: torch.device,
) -> dict[str, torch.Tensor]:
    """Copy the source's storages and rebuild its tensors on them."""
    started = time.perf_counter()
    storages = manifest["storages"]
    buffers = [allocate_storage(storage, device) for storage in storages]
    self.copy_into(storages, seed, buffers)
    tensors = rebuild_tensors(manifest, buffers)
    total_gib = manifest_nbytes(manifest) / 1024**3
    elapsed = time.perf_counter() - started
    logger.info(
        "Mirrored %.2f GiB in %d storage(s) over %s in %.2fs (%.1f GiB/s)",
        total_gib,
        len(buffers),
        self.name,
        elapsed,
        total_gib / max(elapsed, 1e-9),
    )
    return tensors

prepare_seed(storages) abstractmethod

Publish how a mirror can read each storage, in manifest order.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
@abstractmethod
def prepare_seed(self, storages: list[torch.UntypedStorage]) -> dict[str, Any]:
    """Publish how a mirror can read each storage, in manifest order."""

release()

Drop source-side state for the weights the daemon is releasing.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
def release(self) -> None:
    """Drop source-side state for the weights the daemon is releasing."""
    return None

WeightShardSeed

Bases: Seedable

This daemon's weight shard: served to and fetched from peer daemons.

Methods:

  • release –

    Wait for copies, then drop source state without blocking their acks.

  • request_params –

    Ask this daemon's source for the shard, over the fitting mover.

  • resolve –

    Describe this rank so a peer daemon can mirror it.

  • validate_payload –

    Structural checks plus the shard fingerprint against this daemon.

  • validate_remote –

    Refuse node-local movers on the remote control plane.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
class WeightShardSeed(Seedable):
    """This daemon's weight shard: served to and fetched from peer daemons."""

    kind = "weights"

    def __init__(self, daemon: "WeightCacheDaemon"):
        # The daemon this rank serves and fetches for: the manifest, mover
        # seeds and fingerprint are all read from it.
        self._daemon = daemon
        self._movers: dict[SeedBackend, WeightMover] = {}
        self._active_copies = 0
        self._lease_condition = threading.Condition(daemon._state_lock)
        self._release_pending = False

    def _mover(self, backend: SeedBackend) -> WeightMover:
        mover = self._movers.get(backend)
        if mover is None:
            mover = get_mover(backend, device_index=self._daemon.device_index)
            self._movers[backend] = mover
        return mover

    def release(self) -> None:
        """Wait for copies, then drop source state without blocking their acks."""
        deadline = time.monotonic() + WEIGHT_COPY_LEASE_TIMEOUT_S
        with self._lease_condition:
            self._release_pending = True
            if self._active_copies:
                logger.info(
                    "Weight cache %s daemon rank %d waiting for %d "
                    "weight-cache copy acknowledgement(s)",
                    self._daemon.role,
                    self._daemon.global_rank,
                    self._active_copies,
                )
            while self._active_copies:
                remaining = deadline - time.monotonic()
                if remaining <= 0:
                    raise TimeoutError(
                        f"Timed out waiting for {self._active_copies} "
                        "weight-cache copy acknowledgement(s); cached "
                        "weights and NIC registrations were retained"
                    )
                self._lease_condition.wait(remaining)
            # A failed release stays pending: some regions may have been
            # unregistered, so publishing the old bookkeeping is unsafe.
            for mover in self._movers.values():
                mover.release()

    def _acknowledge(self) -> dict[str, bool]:
        with self._lease_condition:
            if not self._active_copies:
                raise ValueError("Unexpected weight-cache acknowledgement")
            self._active_copies -= 1
            self._lease_condition.notify_all()
        return {"acknowledged": True}

    def validate_remote(self, request: dict[str, Any]) -> None:
        """Refuse node-local movers on the remote control plane."""
        if request.get("acknowledge"):
            return
        backend = parse_seed_backend(request.get("seed_backend", PEER_IPC_BACKEND))
        if backend == PEER_IPC_BACKEND:
            raise ValueError(
                f"Seed backend {backend!r} copies through CUDA IPC, which "
                "cannot cross hosts; mirror it over this daemon's Unix socket"
            )

    def resolve(self, request: dict[str, Any]) -> WeightsSeedPayload | dict[str, bool]:
        """Describe this rank so a peer daemon can mirror it."""
        if request.get("acknowledge"):
            return self._acknowledge()
        daemon = self._daemon
        backend = parse_seed_backend(request.get("seed_backend", PEER_IPC_BACKEND))
        with daemon._state_lock:
            if not daemon.has_weights or self._release_pending:
                raise ValueError("Weights were released")
            mover = self._mover(backend)
            tensors, kinds, aliases, attrs = daemon._seed_state()
            manifest, storages = build_manifest(tensors, kinds)
            seed = mover.prepare_seed(storages)
            with self._lease_condition:
                self._active_copies += 1
            return {
                "cache_config": asdict(daemon.cache_key),
                # Matched instead of the checkpoint fingerprint by a mirror
                # without the safetensors to hash (see seed_mismatches).
                "model": daemon.model_config.model,
                "manifest": manifest,
                "aliases": aliases,
                "attrs": attrs,
                "seed": seed,
            }

    def request_params(self, ctx: Any) -> dict[str, Any]:
        """Ask this daemon's source for the shard, over the fitting mover."""
        source = self._daemon.d2d.source
        backend = PEER_IPC_BACKEND if isinstance(source, str) else MOONCAKE_BACKEND
        return {"seed_backend": backend}

    def validate_payload(self, payload: Any) -> WeightsSeedPayload:
        """Structural checks plus the shard fingerprint against this daemon."""
        daemon = self._daemon
        try:
            return validate_weights_payload(
                payload, daemon.cache_key, daemon.model_config.model
            )
        except Exception:
            with contextlib.suppress(Exception):
                acknowledge_lease(daemon.d2d)
            raise

release()

Wait for copies, then drop source state without blocking their acks.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
def release(self) -> None:
    """Wait for copies, then drop source state without blocking their acks."""
    deadline = time.monotonic() + WEIGHT_COPY_LEASE_TIMEOUT_S
    with self._lease_condition:
        self._release_pending = True
        if self._active_copies:
            logger.info(
                "Weight cache %s daemon rank %d waiting for %d "
                "weight-cache copy acknowledgement(s)",
                self._daemon.role,
                self._daemon.global_rank,
                self._active_copies,
            )
        while self._active_copies:
            remaining = deadline - time.monotonic()
            if remaining <= 0:
                raise TimeoutError(
                    f"Timed out waiting for {self._active_copies} "
                    "weight-cache copy acknowledgement(s); cached "
                    "weights and NIC registrations were retained"
                )
            self._lease_condition.wait(remaining)
        # A failed release stays pending: some regions may have been
        # unregistered, so publishing the old bookkeeping is unsafe.
        for mover in self._movers.values():
            mover.release()

request_params(ctx)

Ask this daemon's source for the shard, over the fitting mover.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
def request_params(self, ctx: Any) -> dict[str, Any]:
    """Ask this daemon's source for the shard, over the fitting mover."""
    source = self._daemon.d2d.source
    backend = PEER_IPC_BACKEND if isinstance(source, str) else MOONCAKE_BACKEND
    return {"seed_backend": backend}

resolve(request)

Describe this rank so a peer daemon can mirror it.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
def resolve(self, request: dict[str, Any]) -> WeightsSeedPayload | dict[str, bool]:
    """Describe this rank so a peer daemon can mirror it."""
    if request.get("acknowledge"):
        return self._acknowledge()
    daemon = self._daemon
    backend = parse_seed_backend(request.get("seed_backend", PEER_IPC_BACKEND))
    with daemon._state_lock:
        if not daemon.has_weights or self._release_pending:
            raise ValueError("Weights were released")
        mover = self._mover(backend)
        tensors, kinds, aliases, attrs = daemon._seed_state()
        manifest, storages = build_manifest(tensors, kinds)
        seed = mover.prepare_seed(storages)
        with self._lease_condition:
            self._active_copies += 1
        return {
            "cache_config": asdict(daemon.cache_key),
            # Matched instead of the checkpoint fingerprint by a mirror
            # without the safetensors to hash (see seed_mismatches).
            "model": daemon.model_config.model,
            "manifest": manifest,
            "aliases": aliases,
            "attrs": attrs,
            "seed": seed,
        }

validate_payload(payload)

Structural checks plus the shard fingerprint against this daemon.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
def validate_payload(self, payload: Any) -> WeightsSeedPayload:
    """Structural checks plus the shard fingerprint against this daemon."""
    daemon = self._daemon
    try:
        return validate_weights_payload(
            payload, daemon.cache_key, daemon.model_config.model
        )
    except Exception:
        with contextlib.suppress(Exception):
            acknowledge_lease(daemon.d2d)
        raise

validate_remote(request)

Refuse node-local movers on the remote control plane.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/seed.py
def validate_remote(self, request: dict[str, Any]) -> None:
    """Refuse node-local movers on the remote control plane."""
    if request.get("acknowledge"):
        return
    backend = parse_seed_backend(request.get("seed_backend", PEER_IPC_BACKEND))
    if backend == PEER_IPC_BACKEND:
        raise ValueError(
            f"Seed backend {backend!r} copies through CUDA IPC, which "
            "cannot cross hosts; mirror it over this daemon's Unix socket"
        )

build_manifest(tensors, kinds)

Describe tensors as views over their unique storages.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/utils.py
def build_manifest(
    tensors: Mapping[str, torch.Tensor], kinds: Mapping[str, TensorKind]
) -> tuple[dict[str, Any], list[torch.UntypedStorage]]:
    """Describe `tensors` as views over their unique storages."""
    storages: list[torch.UntypedStorage] = []
    index_by_storage: dict[tuple[torch.device, int], int] = {}
    views: dict[str, dict[str, Any]] = {}
    for name, tensor in tensors.items():
        storage = tensor.untyped_storage()
        index = index_by_storage.setdefault(
            (tensor.device, storage.data_ptr()), len(storages)
        )
        if index == len(storages):
            storages.append(storage)
        views[name] = {
            "storage": index,
            "offset": tensor.storage_offset(),
            "shape": list(tensor.shape),
            "stride": list(tensor.stride()),
            "dtype": str(tensor.dtype).removeprefix("torch."),
            "is_param": kinds[name] == "param",
        }
    manifest = {
        "storages": [
            {"nbytes": storage.nbytes(), "device": storage.device.type}
            for storage in storages
        ],
        "tensors": views,
    }
    return manifest, storages

get_mover(name, *, device_index=None)

The mover for a backend; device_index pins the peer_ipc device.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/core.py
def get_mover(name: SeedBackend, *, device_index: int | None = None) -> WeightMover:
    """The mover for a backend; `device_index` pins the peer_ipc device."""
    if name == PEER_IPC_BACKEND:
        from vllm.model_executor.model_loader.weight_cache.weight_seed.peer_ipc import (
            PeerIpcMover,
        )

        return PeerIpcMover(device_index)
    if name == MOONCAKE_BACKEND:
        from vllm.model_executor.model_loader.weight_cache.weight_seed.mooncake import (
            MooncakeMover,
        )

        return MooncakeMover()
    raise ValueError(f"Unknown weight-cache seed backend {name!r}")

rebuild_tensors(manifest, buffers)

Recreate every manifest tensor on the mirror's copies; bounds-checked first.

Source code in vllm/model_executor/model_loader/weight_cache/weight_seed/utils.py
def rebuild_tensors(
    manifest: Mapping[str, Any], buffers: list[torch.Tensor]
) -> dict[str, torch.Tensor]:
    """Recreate every manifest tensor on the mirror's copies; bounds-checked first."""
    tensors: dict[str, torch.Tensor] = {}
    for name, view in manifest["tensors"].items():
        index = view["storage"]
        if not isinstance(index, int) or not 0 <= index < len(buffers):
            raise RuntimeError(f"Seed manifest entry {name!r} names no storage")
        buffer = buffers[index]
        dtype = manifest_dtype(view["dtype"])
        if _view_extent(view, dtype.itemsize) > buffer.nbytes:
            raise RuntimeError(f"Seed manifest entry {name!r} overruns its storage")
        tensors[name] = torch.empty(0, dtype=dtype, device=buffer.device).set_(
            buffer.untyped_storage(), view["offset"], view["shape"], view["stride"]
        )
    return tensors