Distributed LLM Inference via Communication-Efficient Speculative Decoding

Introduction

Distributed speculative decoding lets a lightweight device collaborate with a powerful server to generate text from large language models (LLMs). The device runs a small draft model, proposes candidate tokens, and the server validates them using the full LLM. The validation step traditionally requires sending the entire probability distribution over the model’s vocabulary from the server back to the device for exact rejection sampling. That vocabulary‑scale communication quickly becomes the bottleneck, especially on mobile or edge links with limited bandwidth.

The paper Distributed LLM Inference via Communication‑Efficient Speculative Decoding proposes DISCO, a framework that eliminates the costly distribution transfer while preserving the exact target distribution. By replicating the draft small language model (SLM) on the server, DISCO enables the server to compute the draft‑side distribution locally. Consequently, only the index of the drafted token and a few scalar probability values need to be exchanged.

How DISCO Works

1. Draft Generation on the Device – The device runs the SLM to produce a candidate token and its associated probability under the draft model.
2. Server‑Side Replica – The server hosts an identical copy of the SLM. When the device sends the drafted token index, the server instantly reconstructs the draft distribution for that step without any additional data from the device.
3. Exact Rejection Sampling – Using the locally available draft distribution and the full‑model distribution (computed on the server), the server performs the standard rejection‑sampling test. If the draft token passes, it is emitted; otherwise, the server draws a token from the full model.
4. Minimal Communication – The only data sent back to the device are the accepted token index and a scalar acceptance probability. No full‑vocab vectors traverse the link.

By moving the draft‑model inference to the server side, DISCO sidesteps the bandwidth‑heavy step that has limited prior speculative decoding approaches.

Performance Highlights

• Speedup Across Bandwidth Regimes – Experiments on several benchmark datasets show that DISCO consistently outperforms earlier distributed speculative decoding methods. The advantage grows as the available bandwidth shrinks, confirming that the communication reduction is the primary gain.
• Preserved Sampling Correctness – Because DISCO still performs exact rejection sampling, the output distribution matches that of the original LLM, unlike approximate methods that trade fidelity for speed.
• Scalability – The framework works with any pair of draft and full models, making it applicable to a range of device capabilities and server‑side LLM sizes.

Practical Implications

For developers building on‑device AI assistants, chatbots, or edge‑deployed language services, DISCO offers a clear path to leverage powerful LLMs without overwhelming the network. The reduced data payload means lower latency, reduced data‑plan costs, and better user experience on constrained connections such as 4G, satellite, or Wi‑Fi‑only environments.

Closing Thoughts

DISCO demonstrates that thoughtful system design—replicating the lightweight draft model on the server—can eliminate a long‑standing communication bottleneck in speculative decoding. By keeping the exact rejection‑sampling guarantees while slashing bandwidth usage, it opens the door for more responsive, cost‑effective LLM inference on a wide variety of devices. Future work may explore adaptive draft‑model selection, compression of the scalar messages, or integration with emerging hardware accelerators to push the speedup envelope even further.