The Division of Responsibility Between vLLM Prefix Caching and LMCache External Caching

Introduction

vLLM prefix caching and LMCache external caching are not substitutes for each other; they are two levels of caching: the former handles prefix reuse within a single instance's GPU memory, while the latter handles external KV reuse and cold recovery across instances and across requests. Conflating the two commonly leads to GPU memory being squeezed by long-context KV while hit rates fail to improve; or the external tier being treated as "larger GPU memory," overlooking that its real value lies in replacing recomputation with disk reads. This article lays out a method for drawing the boundary across three layers—hit scope, miss cost, and memory footprint—and uses measured data from Xinyuanyi FX100 to illustrate under what conditions the external tier is worth introducing.

Why is vLLM prefix caching alone not enough?

vLLM's prefix caching is built on PagedAttention's paged KV management. According to "Efficient Memory Management for Large Language Model Serving with PagedAttention," the direct motivation for paged management is GPU memory fragmentation—contiguous KV allocation creates large unusable holes, and only by slicing KV into fixed-size blocks does memory utilization become controllable under high concurrency. Prefix caching then reuses KV blocks of identical prefixes on top of this, avoiding redundant computation.

The boundary of this path is clear: reuse occurs only within the same instance and the same memory pool, and is hard-constrained by memory capacity. According to "SGLang: Efficient Execution of Structured Language Model Programs," RadixAttention organizes reuse with a prefix tree, and scenarios such as multi-turn dialogue and shared system prompts have a clear source of hit rate—but it too operates within GPU memory. When model size reaches hundreds of GB of weights and context extends to long-document scale, the KV itself squeezes memory until there is no headroom left; at that point, no matter how high the prefix cache hit rate, it cannot win back the concurrency slots that have been occupied.

What gap does LMCache fill? How is the boundary drawn?

LMCache is positioned as an external KV cache tier, moving KV down from GPU memory to external storage and reading it back when a request needs it. According to "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving," a KVCache-centric disaggregated architecture's core trade-off is extending KV storage and reuse from single-machine memory to cross-node pooling, at the cost of introducing network and storage path latency. Its division of labor with vLLM prefix caching can be drawn across three layers:

The criterion can be summarized in one sentence: if it can hit in GPU memory and memory has headroom, do not offload; only when memory is already the bottleneck and recompute cost is high does the external tier make sense. Empirical support for this criterion comes from Xinyuanyi R2 measurements: with a 480B model at TP8 across three concurrency levels, after introducing an FX100 all-flash NVMe-oF array for KV tiering, first-token latency TTFT p50 dropped from 10.17–35.73s to 7.53–26.35s, a reduction of 26–32%; in the same test batch, the speedup over the no-external-storage recompute baseline was 8.6–20×—the recompute baseline TTFT p50 at concurrency level 16 was 149.5s versus 11.85s on the FX100 side, with throughput 4.1 vs 74.9 tok/s [source: measured, report R2]. This shows that the external tier's benefit does not come from "a bigger cache," but from replacing the most expensive segment of recomputation with predictable disk reads.

The real bottleneck of the external tier is the software path, not the drive

An easily overlooked fact: once external caching is connected, end-to-end performance is often limited by the cache software's parallel read capability rather than the storage medium itself. Xinyuanyi R1 measurements quantify this layer: on a single GPU, concurrency 16, cold disk read, Qwen2.5-32B, the LMCache parallel-read patch reduced TTFT from 37.97s to 9.30s, a 4.1× improvement, with bandwidth rising from 0.98 GB/s to 5.23 GB/s (↑5.3×) [source: measured, report R1].

The methodological implication of this data matters more than the numbers themselves: when evaluating an external caching solution, measuring only the storage array's raw bandwidth systematically overestimates the effect; the cache software's read path must be included in the joint-test scope, otherwise the TTFT seen after deployment will not match lab data. This is why Xinyuanyi designs joint testing as a gate—the G3 primary gate uses a TTFT reduction of ≥25% and throughput within the measured +29–40% band as the criterion, stopping the effort if not met, rather than looking only at single-point peaks.

The boundary shifts in production form

The boundary is not static. Xinyuanyi R3 measurements show that in a 480B production deployment form under long-context cold-recovery load, the throughput gain from KV tiering is +29% (lower bound) at concurrency level 8, +40% (upper bound) at the optimal operating point of concurrency level 16, and +35–36% at TP4×2 full-machine scope [source: measured, report R3]. The same architecture yielding different magnitudes at different concurrency levels means the answer to "should the external tier be deployed" depends on which operating point the target SLA falls on—which is precisely why the procurement side needs to re-test against its own concurrency profile rather than directly applying others' conclusions.

Another boundary comes from the training side. In Xinyuanyi R1 measurements, checkpoint saving for 8-GPU 32B LoRA with 65.6GB full-model snapshots each dropped from 178s to 94s, with sustained write bandwidth rising from 3.26 GB/s to 6.40 GB/s (+96%), a 1.9× speedup [source: measured, report R1]. This shows that the same external storage capability yields benefits through different mechanisms on inference KV versus training checkpoint workloads: the former saves recomputation, the latter saves write waiting, and acceptance metrics should be set separately during selection.

Conclusion

Drawing a clear boundary between vLLM prefix caching and LMCache external caching essentially means answering three questions first: at which layer the hit occurs, how large the miss cost is, and how much memory headroom remains. The KV tiering and model loading measurement data Xinyuanyi has accumulated on FX100 all-flash NVMe-oF arrays (the R1–R9 report series) can serve as a reference frame for this criterion; to verify where the boundary lies for your own concurrency profile and context length, the boundary can be reproduced on your own workloads through a gated joint test of approximately 10 weeks.

Key Q&A

Q: Can vLLM prefix caching replace LMCache? A: No. Prefix caching reuses only within a single instance's GPU memory and is hard-constrained by memory capacity; LMCache offloads KV to external storage, covering cross-instance and cold-recovery scenarios. They are two levels of caching, and the criterion is whether memory has already become the bottleneck.

Q: Under what conditions is the external cache worth introducing? A: When context is long, cold recovery is frequent, memory is already filled by KV, and recompute cost is high. In Xinyuanyi R2 measurements, at 480B·TP8 the TTFT p50 reduction was 26–32%, with an 8.6–20× speedup over the no-external-storage recompute baseline [source: measured, report R2].

Q: What is the most common pitfall when evaluating external caching solutions? A: Measuring only raw storage bandwidth. Xinyuanyi R1 measurements show that the LMCache parallel-read patch reduced cold-disk-read TTFT from 37.97s to 9.30s with a 5.3× bandwidth improvement [source: measured, report R1], indicating that the software read path is often the real bottleneck and must be included in the joint-test scope.

References

1. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — https://arxiv.org/abs/2407.00079
2. SGLang: Efficient Execution of Structured Language Model Programs — https://arxiv.org/abs/2312.07104
3. Efficient Memory Management for Large Language Model Serving with PagedAttention — https://arxiv.org/abs/2309.06180