MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Introduction

Mixture‑of‑Experts (MoE) language models achieve impressive scaling by activating only a small subset of experts for each token. While this sparsity reduces compute, the key‑value (KV) cache still grows linearly with the input length during inference, creating a memory bottleneck for long‑context tasks. Existing KV compression techniques focus on dense transformers and ignore a unique issue in MoE models: compressing or evicting KV entries can alter the routing decisions that determine which experts are consulted, potentially harming generation quality.

Routing‑Aware Compression

MoEZip tackles this problem by making KV compression aware of the routing process. The authors propose a sensitivity metric derived from the Fisher information matrix of the routing distributions, which quantifies how much a perturbation to a KV entry would affect expert selection. By combining this metric with traditional attention‑weight scores, MoEZip prioritizes retaining entries that are both influential for attention and critical for stable routing.

Composite Scoring for Dense and Sparse Architectures

Recognizing that MoE models contain both dense and sparse components, MoEZip introduces a complementary scoring system. For dense parts of the network it falls back on established eviction heuristics, while for the sparse MoE layers it applies the routing‑aware scores. This hybrid approach ensures that compression decisions respect the differing dynamics of each sub‑network, avoiding the inadvertent drift of expert assignments that can occur with naïve KV pruning.

Empirical Results

The paper evaluates MoEZip on several long‑context benchmarks, including LongBench and RULER. At a KV retention ratio of just 10 %—meaning 90 % of the cache is discarded—MoEZip outperforms the previous state‑of‑the‑art baseline by roughly 24 points on LongBench and 20 points on RULER on average. These improvements correspond to 2.3× and 4.9× higher scores, respectively, demonstrating that aggressive compression can be achieved without sacrificing, and indeed often improving, generation quality.

Practical Takeaways

• Sensitivity‑Based Scoring: Use the Fisher‑information‑derived metric to gauge how KV compression will impact expert routing.
• Hybrid Eviction Policy: Apply dense‑model eviction strategies to the transformer backbone while reserving routing‑aware scoring for MoE layers.
• Aggressive Compression Viable: Even with only 10 % of KV entries retained, MoEZip maintains or improves performance on long‑context tasks, offering a clear path to lower memory footprints.

Conclusion

MoEZip fills a critical gap in the toolkit for deploying large MoE language models in memory‑constrained environments. By explicitly accounting for routing stability during KV cache compression, it delivers robust accuracy gains at compression levels previously thought impractical. Practitioners looking to run sparse MoE models on long sequences can adopt MoEZip’s routing‑aware scoring to achieve substantial memory savings while preserving, or even enhancing, output quality.