Tokens In <> Tokens Out API¶
POST /inference/v1/generate is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and finish_reason/stop_reason, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the render → generate → derender pipeline used in disaggregated serving (see the Renderer APIs and Derenderer APIs).
Enabling the endpoint (--enable-scale-out, --tokens-only)¶
--enable-scale-outregisters it, with/render,/derenderand/inference/v1/abort_requests.--tokens-onlyalways registers it, skips loading the tokenizer and forcesdetokenize=False. Stop strings (sampling_params.stop) are then rejected with a 400; usestop_token_ids.VLLM_SERVER_DEV_MODE=1turns--enable-scale-outon unless--disable-scale-outis passed.- With neither flag, it returns 403
Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.
vllm serve Qwen/Qwen3-0.6B --enable-scale-out
curl -X POST http://localhost:8000/inference/v1/generate \
-H "Content-Type: application/json" \
-d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
A full client is in examples/scale_out/token_generation_client.py.
Request fields¶
| Field | Notes |
|---|---|
token_ids |
Required list[int]: the already-templated prompt. Non-empty, no negative IDs. |
sampling_params |
Required. Any SamplingParams field, e.g. max_tokens, temperature, logprobs, stop_token_ids. Without max_tokens, the server applies its own default (as on the OpenAI endpoints), not 16. bad_words needs a tokenizer: a server without one (--tokens-only or --skip-tokenizer-init) rejects it with a 400. |
output_mode |
"tokens" (default) or "text", see Output modes. |
stream, stream_options |
SSE chunks ending with data: [DONE]. |
return_token_ids |
Echo the final prompt IDs as prompt_token_ids. |
features / content_parts |
Multimodal input, mutually exclusive. |
request_id, model, cache_salt, priority, kv_transfer_params, ec_transfer_params |
Optional. |
Response fields¶
choices[i] field |
Notes |
|---|---|
token_ids |
Generated token IDs. |
finish_reason |
stop, length or abort. |
stop_reason |
Stop token ID or string; null on EOS, length and abort. |
logprobs |
{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}, one entry per token, sampled token first. token_id is an integer, not the older "token_id:<id>" string. Needs sampling_params.logprobs. |
weight_versions, routed_experts, sampling_mask |
Optional, see Weight versions; the last two need --enable-return-routed-experts / --return-sampling-mask. |
The top level adds request_id, usage, prompt_token_ids and prompt_logprobs. Stream chunks carry the same per-choice fields for the new tokens.
Targeting a data-parallel rank (X-data-parallel-rank)¶
Send the header X-data-parallel-rank: <int> to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside [0, data_parallel_size) is rejected.
Request priority (X-Vllm-Priority)¶
Send the header X-Vllm-Priority: <int> to set the request's priority. It overrides the body priority; a non-integer value is ignored. A header value or body priority outside [-2**31, 2**31 - 1] is rejected. Non-zero priorities require the server to use priority scheduling (--scheduling-policy priority).
Output modes¶
The request field output_mode selects how much of the postprocessing the server does for the caller.
output_mode |
Response adds | Server needs |
|---|---|---|
tokens (default) |
nothing, token IDs only | nothing |
text |
text on every choice and decoded tokens with bytes in logprobs |
a tokenizer |
derender |
the parsed assistant message (content, reasoning, tool_calls) on every choice, plus logprobs shaped like coupled chat's |
a tokenizer and the parser flags you'd give vllm serve |
text returns text and token IDs in a single call, without a separate derender hop per chunk. This suits latency sensitive streaming, where the derender hop lands on every output token.
Every response and every stream chunk echoes output_mode. Servers that predate the field ignore it and return token IDs only with a 200, so check that the output_mode in the response matches the one you sent. Older servers don't return it.
import httpx
response = httpx.post(
"http://localhost:8000/inference/v1/generate",
json={
"token_ids": [151644, 872, 198],
"sampling_params": {"max_tokens": 32},
"output_mode": "text",
},
).json()
assert response["output_mode"] == "text"
print(response["choices"][0]["text"])
print(response["choices"][0]["token_ids"])
Text¶
textcomes from the request's own detokenizer, soskip_special_tokens,spaces_between_special_tokens,stopandinclude_stop_str_in_outputbehave as they do on/v1/completions.- A matched stop string is cut from
textunlessinclude_stop_str_in_outputis set.token_idskeeps every generated token, so decodingtoken_idsyourself brings the stop string back. /derenderonly acceptsoutput_mode: "tokens"responses and returns a 400 for text responses which are already detokenized.- With
stream: true, each choice'stextis the delta since the previous chunk. A chunk is sent whenever the engine output carries new text or afinish_reason, even with no new token IDs. That covers text held back for stop string matching and the final output after an abort.
Derender¶
derender returns the assistant message a coupled /v1/chat/completions would, without a separate derender call. /v1/chat/completions/render and /v1/messages/render return parse_context on the GenerateRequest. Send it back with output_mode: "derender". It carries the tools, tool_choice, include_reasoning, return_tokens_as_token_ids, resolved JSON only chat_template_kwargs and the history tool call count. A parse_context field this server doesn't know fails a derender request and is ignored at the other levels.
- The parser reads the engine's
text, so it sees the stop string cut and the postadjust_requestdecode flags. That makes it match coupled chat for parsers whoseadjust_requestchanges decoding which/derendercan't. - A server with a tool or reasoning parser needs
parse_context. Without a parser,message.contentis the plain text. parse_contextis ignored at other levels, so a rendered request can go to a prefill leg unchanged.logprobsfollow coupled chat.top_logprobsis cut tosampling_params.logprobs,logprob_token_idsis honored. When a parser hides the reasoning withinclude_reasoning: false,logprobs,token_idsandsampling_maskare dropped so they don't expose it.finish_reasonfollows coupled chat too. It'stool_callswhen the message has tool calls except with a namedtool_choiceor alengthcut.- With
stream: true, each choice carries a chatdelta(role,content,reasoning,tool_calls) instead oftext, andfinish_reasonfollows coupled chat.roleis set on the first chunk of each choice,n > 1is supported, andnullfields are left out ofdelta. A chunk is sent whenever the engine output carries a parsed delta, visible token IDs, logprobs or afinish_reason. Parsing runs on the API server event loop, so--api-server-countscales it. - The message
roleis the generate server's--response-role.parse_contextdoesn't carrymessages,add_generation_promptorecho, so the last message's role andechoaren't applied. Use/derenderfor those. - Cohere grounding citations aren't returned. The parser strips the citation spans from
content, and the document map can't go over JSON inparse_context.
A streamed tool call looks like this, with the usage chunk and data: [DONE] framed as at the other levels:
data: {"request_id": "...", "output_mode": "derender", "choices": [{"index": 0, "token_ids": [151657], "delta": {"role": "assistant", "tool_calls": [{"index": 0, "id": "chatcmpl-tool-...", "type": "function", "function": {"name": "get_weather", "arguments": ""}}]}, "finish_reason": null, ...}]}
data: {"request_id": "...", "output_mode": "derender", "choices": [{"index": 0, "token_ids": [], "delta": {}, "finish_reason": "tool_calls", ...}]}
Logprobs¶
With output_mode: "tokens", logprobs are GenerateLogProbs with an integer token_id and rank per entry and no token or bytes (see Generate Output Logprobs). With output_mode: "text", they are ChatCompletionLogProbs carrying the decoded token strings and their UTF-8 bytes for the sampled token and every entry in top_logprobs; a server started with --return-tokens-as-token-ids returns token_id:N placeholders there instead, as /v1/completions does.
With output_mode: "derender", parse_context.return_tokens_as_token_ids overrides the generate server's --return-tokens-as-token-ids when it is not null: true returns token_id:N placeholders with bytes: null, and false returns decoded tokens and UTF-8 bytes even when the server flag is enabled. An omitted or null value uses the generate server's flag. /render preserves explicit true and false without applying the render server's flag, and omits the context field when the request value is omitted or null. This setting affects logprobs only; the assistant message is still parsed from text.
Upgrade generate nodes before using explicit request overrides: older generate nodes reject parse_context.return_tokens_as_token_ids with a 400 in derender mode. Requests with an omitted or null override remain compatible because /render leaves that field out. Unknown parse-context fields remain ignored in tokens and text modes.
Errors¶
The server returns a 400 for:
output_mode: "text"or"derender"on a server without a tokenizer (--tokens-onlyor--skip-tokenizer-init).output_mode: "text"or"derender"withsampling_params.detokenize: false.output_mode: "derender"withoutparse_contexton a server with a tool or reasoning parser or with a namedtool_choiceinparse_contextthat matches none of itstools. This is checked before generating.- An unsupported
output_modevalue.
For using output_mode with separate prefill and decode pools, see Disaggregated Prefilling.
Weight versions¶
The final output of every choice carries weight_versions, the label of the weights that sampled each range of its output tokens. Ranges are half-open over output-token indexes (prompt excluded), contiguous and in generation order, so a request that crossed one weight update gets two:
"weight_versions": [
{"version": "v7", "start": 0, "end": 57},
{"version": "v8", "start": 57, "end": 128}
]
- With
stream: true, only the last chunk of each choice carries the list, including the last chunk after an abort. Its offsets cover the whole output. Earlier chunks carrynullor omit the field. - An output with no tokens, such as a request aborted before its first token, gets
[]. - The labels are the ones you set with
/update_weight_versionor/finish_weight_update, see Async Reinforcement Learning. vLLM doesn't order or check them, and the label is"default"until you set one.
Multimodal inputs¶
A request can carry media in two ways, and can use both at once:
content_parts: raw media (image_url,audio_urlorvideo_url, each with an optionaluuid).token_idsholds one placeholder token per item, and the server expands it.features: items whose placeholders are already expanded intoken_ids, at the ranges given inmm_placeholders. Pass the processed tensors inkwargs_data, or omit them to look the items up in the processor cache bymm_hashes.
When both are set, every content_parts placeholder must come after the last features range. Only the tokens after that range are processed, so the expanded runs before it are left as they are. Models that insert media at the start of the prompt (such as PaliGemma, BLIP-2 and Molmo) would place a new item right after that range without an error, so don't combine the two fields for them.
This fits multi-turn rollouts. Each turn resends the previous prompt with its images already expanded, adds the new turn with one placeholder per new image, and sends the earlier images as features:
import httpx
url = "http://localhost:8000/inference/v1/generate"
# Turn 1: turn1_ids holds one <|image_pad|> for image A.
turn1 = httpx.post(url, json={
"token_ids": turn1_ids,
"content_parts": [{"type": "image_url", "url": url_a, "uuid": "img-a"}],
"sampling_params": {"max_tokens": 256},
"return_token_ids": True,
}).json()
# Turn 2: image A is expanded in turn1["prompt_token_ids"];
# turn2_ids appends one <|image_pad|> for image B.
turn2 = httpx.post(url, json={
"token_ids": (
turn1["prompt_token_ids"] + turn1["choices"][0]["token_ids"] + turn2_ids
),
"features": {
"mm_hashes": {"image": ["uuid:img-a"]},
"mm_placeholders": turn1["mm_placeholders"],
},
"content_parts": [{"type": "image_url", "url": url_b, "uuid": "img-b"}],
"sampling_params": {"max_tokens": 256},
"return_token_ids": True,
}).json()
Here "uuid:img-a" finds image A in the processor cache because turn 1 sent it with uuid "img-a". The cache key of an item sent with a uuid is that uuid prefixed with uuid:, which keeps it apart from the content hashes vLLM computes. Two cases need a different approach:
- If the server sets
--mm-processor-kwargsor--media-io-kwargs, theuuidis hashed with them and is no longer the cache key. Takefeaturesfrom a render response instead. - If the item has been evicted from the cache, the request fails. Resend the item with
kwargs_data.
Aborting requests¶
POST /inference/v1/abort_requests aborts in-flight requests. It is registered wherever /inference/v1/generate is and requires the API key when --api-key is set, like /inference/v1/generate.
curl -X POST http://localhost:8000/inference/v1/abort_requests \
-H "Authorization: Bearer $VLLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{"request_ids": ["generate-tokens-42"]}'
The request ID is the request_id the server returned: the X-Request-Id header or body request_id you sent, prefixed with generate-tokens-. A missing request_ids returns a 400. The response is empty and the abort finishes in the background.
With --tokens-only, the same handler is also served at POST /abort_requests for existing deployments. That path doesn't require the API key, even when --api-key is set. See Security.