What a Weight Update Buys: Leverage Across Training Stages

What a Weight Update Buys: Leverage Across Training Stages

Leverage—the Fisher‑quadratic form of a weight update normalized by random directions—offers a precise lens for comparing how much a single parameter change actually moves a language model toward better performance. By applying this metric to a range of publicly released checkpoints, the authors expose dramatic shifts in efficiency from pretraining through various fine‑tuning regimes.

---

1. Defining Leverage

Leverage is computed in two steps:

1. Signal measurement – run a forward‑mode pass for the update direction to obtain the Fisher quadratic form, which captures how the loss curvature aligns with the update.
2. Normalization – divide by the same quadratic form evaluated on random directions of identical norm at the same checkpoint.

The resulting ratio tells us how many times more “useful” a given update is compared to a random perturbation. A leverage of 1 means the update is no better than noise; higher values indicate stronger alignment with the loss landscape.

---

2. Leverage Across Training Phases

Applying leverage to several checkpoints (Olmo‑3, Tülu‑3, Zephyr, OLMo‑2) reveals a clear hierarchy:

• Pretraining windows (30 k–1 M steps) sit near the random baseline on prose tasks, confirming that most early‑stage weight changes act like noise.
• Supervised fine‑tuning (SFT) pushes leverage roughly fivefold above random, showing that task‑specific data begins to steer updates effectively.
• Reinforcement learning stages (RL‑Zero, RLVR) achieve leverage in the range 400–1 500, indicating a substantial efficiency gain.
• Direct Preference Optimization (DPO) spikes even higher, with leverage values between 400 and 23 000, suggesting that preference‑based signals can extract far more value per parameter update.

These findings support a narrative where the quality of an update, not just its magnitude, evolves dramatically as training progresses.

---

3. The Role of Training Precision

The paper also investigates how numerical precision influences leverage. Training with bf16 tensors without an fp32 master copy freezes a large fraction (36‑99 %) of weight coordinates, dramatically reducing leverage—up to sevenfold in some cases. Moreover, the remaining active coordinates show a strong correlation between update magnitude and the diagonal of the Fisher matrix, a pattern absent in full‑precision fp32 training.

This suggests that low‑precision regimes can inadvertently bias updates toward directions that are already well‑aligned with the loss curvature, limiting the model’s ability to explore new, potentially beneficial directions.

---

4. Predicting the Cost of Random Drop‑and‑Rescale (DARE)

A theoretical contribution of the work is a simple law for the sign‑flip null: the relative degradation caused by randomly dropping and rescaling a fraction $p$ of the update is predicted as

``` (1‑p) / (p * ρ_sign) ```

where $ρ_{\text{sign}}$ measures the alignment of update signs with the Fisher. Empirically, this formula stays within a factor of two of observed damage across a wide range of DARE experiments, providing a practical tool for estimating how much robustness is lost when updates are sparsified.

---

5. Practical Takeaways

• Measure before you prune: Leverage offers a cheap, single‑pass diagnostic to decide whether a proposed weight change is worth keeping.
• Fine‑tune with purpose: Moving from pretraining to SFT and RL stages dramatically improves leverage; designers should allocate compute accordingly.
• Watch precision: bf16 training without an fp32 master can cripple leverage; maintaining a high‑precision master copy is advisable for maximal efficiency.
• Use the sign‑flip null: When experimenting with sparse or noisy update schemes, the derived law gives a quick sanity check on expected performance loss.

---

Closing Thoughts

By grounding the abstract notion of “how much a weight update buys” in a concrete, measurable quantity, leverage bridges theory and practice. It clarifies why early pretraining appears noisy, why later fine‑tuning stages are far more effective, and how implementation details like numerical precision can make or break training efficiency. Researchers and engineers can adopt leverage as a diagnostic tool to better allocate compute, design training pipelines, and evaluate novel update strategies.