Your Experts Have Partners. There's HOPE for Keeping Them Together

HOPE improves expert pruning in Mixture‑of‑Experts models by accounting for pairwise expert interactions, yielding better performance especially at high pruning rates.

Your Experts Have Partners. There's HOPE for Keeping Them Together

Higher‑Order Pruning of Experts (HOPE) – A Practical Overview

Mixture‑of‑Experts (MoE) language models achieve massive scale by routing each token to a small subset of experts. While this design is memory‑efficient during training, at inference time every expert’s parameters must still reside in memory, even though most remain idle. Expert pruning—permanently removing the least useful experts—offers a simple way to cut memory and latency without additional training. The challenge is deciding which experts can be safely removed.

Why First‑Order Pruning Falls Short

Traditional pruning methods assign each expert a scalar importance score based on a calibration dataset and prune the lowest‑scoring experts independently. This first‑order view assumes expert contributions are independent, ignoring the fact that experts often form cooperative groups that are repeatedly co‑activated. Removing an expert that participates in a strong group can disproportionately degrade performance, especially when pruning aggressively (e.g., 40‑50%).

The HOPE Idea: Capture Pairwise Interactions

HOPE (Higher‑Order Pruning of Experts) treats pruning as a quadratic optimization problem. For a candidate prune‑set \(P\), the method estimates the expected pruning error using a surrogate \(Z\) that includes cross‑terms between experts:

\[ Z = \Bigl[\sum{j \in P \cap T(x)} gj(x) \|fj(x)\|2 \Bigr]^2 \]

Expanding the square yields terms of the form \(gi(x)gj(x)\|fi(x)\|2\|fj(x)\|2\), which directly measure pairwise cooperation. These interactions are collected into an \(F\) matrix:

\[ F{i,j} = \frac{1}{|\mathcal{X}{i,j}|}\sum{x \in \mathcal{X}{i,j}} gi(x)gj(x)\|fi(x)\|2\|fj(x)\|2 \]

where \(\mathcal{X}{i,j}\) are tokens that activate both experts \(i\) and \(j\). The diagonal \(F{k,k}\) recovers the familiar first‑order importance scores (essentially the REAP ranking), while the off‑diagonal entries capture the co‑activation strength between expert pairs.

HOPE then solves:

\[ \min{p \in \{0,1\}^E}\; p^{\top}Fp \quad \text{s.t.}\; \sumk p_k = |P| \]

using a continuous relaxation followed by rounding. The result is a prune‑set that minimizes the expected error while preserving high‑interaction groups.

Running HOPE on Your Model

  1. Prepare a calibration dataset – any representative text corpus (e.g., code snippets for coding models). One forward pass over this data is required.
  2. Collect gate weights and expert outputs for each token to build the \(F\) matrix per MoE layer.
  3. Solve the quadratic program – standard QP solvers handle each layer in 1–2 seconds; the overhead is negligible compared to the calibration pass.
  4. Prune the selected experts and optionally fine‑tune or quantize the resulting model.

The workflow mirrors existing first‑order pruning pipelines, making adoption straightforward.

When Does HOPE Shine?

Empirical evaluations on models such as Qwen‑3.5‑122B, Qwen‑3.5‑35B, and GLM‑4.5‑Air show that HOPE consistently achieves the best average rank across diverse benchmarks (agentic coding, reasoning, etc.). Its advantage grows with pruning intensity:

  • At 30 % pruning, HOPE matches or slightly exceeds the next‑best method.
  • At 50 % pruning, HOPE’s average rank improves to 1.58 versus 2.42 for REAP, the strongest first‑order baseline.
  • On the SWE‑Bench Pro coding benchmark, a 50 % pruned Qwen‑3.5‑35B model using HOPE attains a score 42.3, essentially matching the unpruned model (41.8) while REAP drops to 36.2.

These gains are most pronounced on tasks that require diverse expert combinations, such as multi‑turn tool‑use or code generation, where preserving cooperative structures is critical.

Limitations and Practical Tips

  • Calibration quality matters – the \(F\) matrix reflects the data seen during calibration. Use a dataset that mirrors the target workload.
  • Quadratic program scaling – the matrix size grows with the number of experts per layer. For very large expert counts, consider block‑wise approximations or low‑rank factorizations.
  • No training required – HOPE is a post‑training step, but you may still benefit from a brief fine‑tune after pruning to recover any minor degradation.

Bottom Line

HOPE offers a second‑order, interaction‑aware approach to expert pruning that retains the simplicity of existing methods while delivering measurable performance improvements, especially when aggressive memory savings are needed. By incorporating pairwise expert cooperation into the prune‑set selection, HOPE helps keep the most valuable expert groups intact, enabling smaller, faster MoE models without sacrificing capability.

Source

Read original source

Why we picked this

Hugging Face content likely about AI partnerships or collaborations, relevant but not core.

← Back to all articles