Close the HTTP connection. In Sorami’s tokens-after-cancel study on vLLM 0.31.0, a transport close stopped generation at the abandon point in every arm tested, while a client that kept reading or held the socket unread left the engine running. Close each timed-out attempt before retrying, test your gateway, and log engine-side terminals.
The journals, plans and analysis are in Sorami’s public data repository. This guide takes only steps the report supports: one Kubernetes cluster in AWS and six fresh-engine blocks per arm, with synthetic prompts and greedy decoding. Engines, versions and gateways that were not run are not covered.
Key takeaways
- A transport close left 0.0 accepted tokens after the close on vLLM 0.31.0.
- Draining accepted 52.8 more tokens per abandoned request; an unread socket 44.3 more.
- LiteLLM 1.105.1 passed the close through; Bifrost transports/v2.3.0 added about 3 tokens.
- SGLang 0.5.21 let 2.0 tokens through after a close; an unread socket stayed alive.
What did the study measure?
An in-process observer on the engine scheduler counted scheduler-accepted tokens, not what the client saw. Three client behaviours were compared after the same abandon point: a transport close, a drained walk-away (keep reading to the end) and a bounded no-read (hold the socket unread for 5 s and then close). Each ran under abandonment after 8 chunks and under a 1.5 s attempt deadline with retry.
On vLLM 0.31.0 with Qwen3-8B on one NVIDIA L4, the close left 0.0 tokens after the close; under retries, draining left 147.9 orphan tokens per task against 0.0 for the close. The same direction held for gpt-oss-20b on L4 and A10G, at batch 64, on free-form prompts and under tensor and pipeline parallelism. gpt-oss-20b was not run in parallel.
How do you make a cancel reach the engine?
1. Close the connection; stopping the read is not a cancel
The engine learns about a cancel only when the transport closes, so the stop button must close the HTTP connection. In Python that is stream.close() or leaving the stream’s context manager (openai-python); in Node, breaking out of the for await loop aborts the request (openai-node); with fetch, pass an AbortSignal and call abort(). Stopping the read while leaving the socket open is the unread-socket condition: 44.3 more tokens per request than a close on vLLM, and 155.1 orphan tokens per set on SGLang under retries.
with client.chat.completions.create(model=MODEL, messages=msgs, stream=True) as stream:
for chunk in stream:
if user_cancelled():
break # leaving the block calls stream.close()2. Close every attempt you replace before you retry
A timed-out attempt that was not closed kept decoding beside its replacement. On one L4, closing the stale attempt took completion within the deadline from 20.7% (drained) and 11.3% (no-read) to 38.0%. Abort the first attempt’s connection, then send the retry.
3. Test the close through your gateway
LiteLLM 1.105.1 passed the client close through with no extra engine tokens, and its 1 s timeout stopped the upstream under every client condition. Bifrost transports/v2.3.0 closed the upstream about 3 tokens later, and its 1 s timeout did not stop the upstream unless the client closed. A 502 after engine admission doubled engine work on both gateways (61.95 orphan tokens per set). Run the harness against your own gateway before trusting either number.
4. Give SGLang requests a server-side deadline
SGLang 0.5.21 detects a disconnect when its next write fails and polls a wait queue with a 4 s timeout (tokenizer manager source). In the study that meant 2.0 tokens after a close and a 120 ms terminal. Under retries the close-to-terminal mean was 2,300 ms, unexplained in the report, and the unread socket did not stop retried requests. Add your own deadline.
5. Log engine-side terminals and alert on late ones
Client token counts miss everything produced after the client looks away, and reasoning models produce hidden tokens before the first visible chunk. Log each request’s terminal status and accepted token count at the engine, matched to the client close time. Tokens accepted after the close are the orphan signal.
6. Treat hosted metering as a dated observation
On 7 October 2026, with one prompt shape, hanging up stopped the meter short of the full answer on every OpenRouter and Amazon Bedrock route tested. Those are provider-reported usage figures, not an invoice, and they were not rerun. Check your provider’s generation records against delivered text yourself.
Sorami’s view
This section is opinion. Most orphan generation we measured came from client and gateway behaviour, not the engine, so fix the stop button and the retry path first. Pick a gateway on measured cancel behaviour, and keep a server-side deadline on SGLang until its queue behaviour under load is explained. Request cancellation belongs beside rate limits and timeouts in an AI production readiness review; if serving spend is the question, a cloud cost review looks at the GPU hours around it.
Checklist
- Close the HTTP connection on cancel; stopping the read is not a cancel.
- Abort each timed-out attempt before sending its retry.
- Test the close and the timeout through your gateway with the harness.
- Abort upstream when the gateway returns an error for an admitted request.
- Set a server-side deadline on SGLang requests under load.
- Log engine-side terminals with status and token count, matched to the close.
- Count reasoning-model tokens at the engine, not from client chunks.
Related reading
The full technical report has the method, tables, figures and limits. For the network side of the same stacks, see how to secure Ray and vLLM on Kubernetes. More is in the guides index.
Sources
- Sorami Technical Report: tokens after cancel on vLLM 0.31.0, SGLang 0.5.21 and two gateways
- Data and evidence repository: observer journals, plans, analysis and harness
- openai-python: Stream.close() in src/openai/_streaming.py
- openai-node: docs/streaming.md, breaking out of a for await loop aborts the request
- SGLang v0.5.21: TokenizerManager._stream_one_response
- LiteLLM v1.105.1 release
- Bifrost transports/v2.3.0 release