Skip to content

vllm.distributed.kv_transfer.kv_connector.v1.mooncake.transfer.backend

Classes:

BatchTransferResult dataclass

Terminal outcome covering num_ranges, not a count of successful ranges.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@dataclass(frozen=True, slots=True)
class BatchTransferResult:
    """Terminal outcome covering `num_ranges`, not a count of successful ranges."""

    num_ranges: int
    state: TransferState
    error: str | None = None

TransferBackend

Bases: ABC

Moves KV blocks between two workers synchronously.

Methods:

Attributes:

  • rpc_port (int) –

    Port peers use to reach this worker's engine.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
class TransferBackend(ABC):
    """Moves KV blocks between two workers synchronously."""

    @property
    @abstractmethod
    def rpc_port(self) -> int:
        """Port peers use to reach this worker's engine."""

    @abstractmethod
    def register_regions(self, regions: Sequence[MemoryRegion]) -> None:
        """Pins local memory for transfers.

        An empty sequence is a no-op. Raises RuntimeError when the engine
        rejects the memory.
        """

    @abstractmethod
    def unregister_regions(self, regions: Sequence[MemoryRegion]) -> None:
        """Releases memory pinned by `register_regions`.

        An empty sequence is a no-op.
        """

    @abstractmethod
    def transfer(
        self, peer: str, op: TransferOp, ranges: Sequence[RangeRequest]
    ) -> list[TransferEvent]:
        """Moves every range and returns one terminal event for each.

        The result lines up with `ranges` by position, and an empty request
        returns an empty list. Blocks until every range reached a terminal
        state. Called from worker threads, so an implementation must accept
        concurrent calls.
        """

    @abstractmethod
    def transfer_sync_arrays(
        self,
        peer: str,
        op: TransferOp,
        local_addrs: NDArray[np.uint64],
        remote_addrs: NDArray[np.uint64],
        lengths: NDArray[np.uint64],
    ) -> BatchTransferResult:
        """Moves equal-length, 1D, contiguous native uint64 columns.

        Blocks until all ranges are terminal, holding the input buffers for the
        duration of the call. Callers must not modify them meanwhile. Concurrent
        calls keep their own inputs and results. Empty input completes without
        engine work. COMPLETED means all ranges succeeded; FAILED does not mean
        all failed. Use the first failure in input order when available.

        Returns one batch summary without constructing per-range requests,
        events or status arrays. Invalid inputs and engine exceptions propagate.
        """

    @abstractmethod
    def shutdown(self) -> None:
        """Stops the backend.

        The caller stops submitting before calling this, but work already
        queued on its worker threads is not cancelled, so a `transfer` call may
        still be running or start afterwards. An implementation must tolerate
        that.
        """

rpc_port abstractmethod property

Port peers use to reach this worker's engine.

register_regions(regions) abstractmethod

Pins local memory for transfers.

An empty sequence is a no-op. Raises RuntimeError when the engine rejects the memory.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@abstractmethod
def register_regions(self, regions: Sequence[MemoryRegion]) -> None:
    """Pins local memory for transfers.

    An empty sequence is a no-op. Raises RuntimeError when the engine
    rejects the memory.
    """

shutdown() abstractmethod

Stops the backend.

The caller stops submitting before calling this, but work already queued on its worker threads is not cancelled, so a transfer call may still be running or start afterwards. An implementation must tolerate that.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@abstractmethod
def shutdown(self) -> None:
    """Stops the backend.

    The caller stops submitting before calling this, but work already
    queued on its worker threads is not cancelled, so a `transfer` call may
    still be running or start afterwards. An implementation must tolerate
    that.
    """

transfer(peer, op, ranges) abstractmethod

Moves every range and returns one terminal event for each.

The result lines up with ranges by position, and an empty request returns an empty list. Blocks until every range reached a terminal state. Called from worker threads, so an implementation must accept concurrent calls.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@abstractmethod
def transfer(
    self, peer: str, op: TransferOp, ranges: Sequence[RangeRequest]
) -> list[TransferEvent]:
    """Moves every range and returns one terminal event for each.

    The result lines up with `ranges` by position, and an empty request
    returns an empty list. Blocks until every range reached a terminal
    state. Called from worker threads, so an implementation must accept
    concurrent calls.
    """

transfer_sync_arrays(peer, op, local_addrs, remote_addrs, lengths) abstractmethod

Moves equal-length, 1D, contiguous native uint64 columns.

Blocks until all ranges are terminal, holding the input buffers for the duration of the call. Callers must not modify them meanwhile. Concurrent calls keep their own inputs and results. Empty input completes without engine work. COMPLETED means all ranges succeeded; FAILED does not mean all failed. Use the first failure in input order when available.

Returns one batch summary without constructing per-range requests, events or status arrays. Invalid inputs and engine exceptions propagate.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@abstractmethod
def transfer_sync_arrays(
    self,
    peer: str,
    op: TransferOp,
    local_addrs: NDArray[np.uint64],
    remote_addrs: NDArray[np.uint64],
    lengths: NDArray[np.uint64],
) -> BatchTransferResult:
    """Moves equal-length, 1D, contiguous native uint64 columns.

    Blocks until all ranges are terminal, holding the input buffers for the
    duration of the call. Callers must not modify them meanwhile. Concurrent
    calls keep their own inputs and results. Empty input completes without
    engine work. COMPLETED means all ranges succeeded; FAILED does not mean
    all failed. Use the first failure in input order when available.

    Returns one batch summary without constructing per-range requests,
    events or status arrays. Invalid inputs and engine exceptions propagate.
    """

unregister_regions(regions) abstractmethod

Releases memory pinned by register_regions.

An empty sequence is a no-op.

Source code in vllm/distributed/kv_transfer/kv_connector/v1/mooncake/transfer/backend.py
@abstractmethod
def unregister_regions(self, regions: Sequence[MemoryRegion]) -> None:
    """Releases memory pinned by `register_regions`.

    An empty sequence is a no-op.
    """