To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems

Introduction

Disaggregated large‑language‑model (LLM) serving splits the prefill and decode stages onto separate node pools, inserting a high‑speed network fabric between the KV‑cache producer and consumer. In this architecture a cache miss no longer means only local recomputation; it also incurs a costly inter‑node transfer. Traditional cache policies such as LRU therefore become inefficient, wasting both memory and bandwidth when they retain blocks that are unlikely to be reused.

The paper To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems proposes KVLearn, a learning‑based framework that makes retention decisions aware of both reuse probability and the concrete costs of recomputation, transfer, and storage. KVLearn combines three components—a lightweight Prefix Reuse Predictor, a Cost‑Aware Retention Score, and an Adaptive Threshold Controller—to achieve substantial reductions in end‑to‑end latency and network traffic while staying close to an oracle’s throughput.

---

1. Prefix Reuse Predictor (PRP)

The PRP estimates the likelihood that a KV‑cache block will be needed again during the decode phase. It extracts structural features (e.g., token type, position in the prompt) and temporal features (e.g., time since the block was created, recent hit patterns) without altering the underlying model weights. A shallow neural network or gradient‑boosted tree can be trained on historical serving logs, learning to map these features to a reuse probability. Because the predictor is lightweight, it can run on the prefill node with negligible overhead, delivering a probability score for each newly generated KV block.

Key advantages:

• Model‑agnostic – works with any transformer‑based LLM.
• Fast inference – sub‑millisecond per block, suitable for high‑throughput pipelines.
• Online adaptability – periodic fine‑tuning incorporates evolving workload characteristics.

---

2. Cost‑Aware Retention Score (CARS)

Raw reuse probability alone does not capture the trade‑off between keeping a block locally and sending it across the network. CARS converts the probability into a retention score by weighting three cost dimensions:

1. Recomputation cost – CPU/GPU cycles needed to regenerate the block if it is evicted.
2. Transfer cost – bandwidth and latency incurred when moving the block to the decode node.
3. Storage cost – memory pressure on the prefill pool, expressed as a penalty for occupying scarce slots.

The formula is a simple linear combination: ``` CARS = preuse * (Crecomp + Ctransfer) - λ * Cstorage ``` where `p_reuse` is the PRP output and `λ` balances memory pressure against latency savings. A positive CARS indicates that keeping the block is expected to reduce overall serving cost, while a negative score suggests eviction or delayed admission.

---

3. Adaptive Threshold Controller (ATC)

Retention decisions are made by comparing each block’s CARS to a dynamic threshold. The ATC monitors two real‑time signals:

• Observed hit rate – the fraction of admitted blocks that later produce a hit.
• Memory pressure – the ratio of used to total KV‑cache memory on the prefill node.

If hit rates fall below a target, the ATC lowers the threshold, allowing more blocks to be admitted. Conversely, when memory usage spikes, the threshold rises, tightening admission criteria. This feedback loop runs continuously, ensuring the system adapts to workload bursts (e.g., long‑form generation) and periods of low reuse (e.g., single‑turn queries).

---

4. Empirical Impact

Integrating KVLearn into a globally disaggregated serving topology yields the following practical benefits across text‑only and multimodal workloads:

• Latency reduction – up to 56 % lower time‑to‑first‑token compared with a no‑cache baseline, and 38 % versus an LRU‑pool policy.
• Network traffic savings – inter‑node KV transfers drop by as much as 53 % relative to LRU‑pool, freeing bandwidth for other services.
• Throughput stability – on the MM‑Session benchmark, overall throughput stays within ~5 % of an oracle that knows future reuse perfectly.

These gains stem from KVLearn’s ability to keep only the most valuable KV blocks, avoiding unnecessary transfers and recomputation while respecting memory constraints.

---

Practical Takeaways

1. Learned reuse prediction is feasible without modifying the LLM itself; lightweight feature‑based models suffice.
2. Cost‑aware scoring bridges the gap between probability and system‑level impact, turning abstract reuse likelihood into concrete admission decisions.
3. Online threshold adaptation is essential in disaggregated environments where workload characteristics and memory pressure fluctuate rapidly.
4. Deploying KVLearn requires modest engineering effort: add the PRP on the prefill side, compute CARS, and run the ATC loop to adjust thresholds. Existing cache infrastructure can be reused for eviction and admission.

---

Conclusion

KVLearn demonstrates that intelligent, cost‑sensitive cache retention can dramatically improve the efficiency of disaggregated LLM serving. By predicting prefix reuse, quantifying the true cost of keeping versus moving KV blocks, and continuously tuning admission thresholds, systems can achieve lower latency, reduced network load, and near‑optimal throughput without sacrificing model accuracy. As LLM deployments continue to scale across heterogeneous hardware pools, such learning‑driven cache policies will become a cornerstone of high‑performance, cost‑effective inference pipelines.