llm-d's precise index never sees the decode pod's reply
The precise index hashes in 64-token blocks and vLLM reports decode-time KV 16 tokens at a time, so the router never learns the decode pod holds the reply. On Qwen3 th...
2026-10-28 · 7 minDeep, specific write-ups on the infrastructure beneath large language models — distributed inference, prompt caching, and the telemetry that makes it debuggable. Read the code, run it on real GPUs, report what actually happens.
The precise index hashes in 64-token blocks and vLLM reports decode-time KV 16 tokens at a time, so the router never learns the decode pod holds the reply. On Qwen3 th...
2026-10-28 · 7 minvLLM's KV-event stream never fires on a cache hit, and the caches under it aren't the radix-trie-plus-RCU design everyone whiteboards. Two corrections for anyone build...
2026-07-15 · 6 minSplit vLLM into prefill/decode under Dynamo and the KV-events subject moves from component.backend to component.prefill. A router still on the old subject gets zero ev...
2026-07-15 · 5 minvLLM's V1 engine, one component at a time — request in, token out, read from the source rather than the paper.