Moment-Guided Optimizers for Noisy Deep Learning: MGO-SC as a Simple Consistency Gate for Adam-Style Updates

Introduction

Adaptive optimizers such as Adam and AdamW dominate modern deep‑learning pipelines, but their performance can suffer when gradients become highly stochastic—common in settings with label noise or small batch sizes. The paper Moment‑Guided Optimizers for Noisy Deep Learning: MGO‑SC as a Simple Consistency Gate for Adam‑Style Updates proposes a lightweight modification that leverages the stability of gradient signs to modulate per‑coordinate updates. By integrating this gate into the Adam update rule, the authors aim to retain the efficiency of first‑order methods while gaining resilience to noisy training signals.

How the Consistency Gate Works

MGO‑SC builds on the exponential moving average (EMA) of gradient signs that Adam already maintains. For each parameter coordinate, the magnitude of this sign‑EMA reflects how consistently the gradient points in the same direction over recent steps. The gate operates as follows:

1. Compute sign‑EMA magnitude – a value between 0 and 1 indicating sign stability.
2. Apply a coordinate‑wise mask – stable coordinates receive a weight close to 1, while fluctuating ones are down‑weighted.
3. Update the Adam step – the original AdamW update is multiplied by the mask, effectively trusting only the reliable directions.

The authors note that a naïve gate that only shrinks updates can reduce the overall step size, harming convergence when the learning rate is matched to the baseline optimizer. To address this, they introduce a mean‑preserving variant that redistributes trust across coordinates without changing the net per‑tensor magnitude of the update. This reallocation preserves the effective learning rate while still damping noisy directions.

Empirical Findings

The paper evaluates MGO‑SC on CIFAR‑10 using a ResNet‑18 architecture under various label‑noise regimes (clean, symmetric, and asymmetric). Experiments keep the learning‑rate schedule and weight‑decay settings identical to a strong AdamW baseline, ensuring a fair comparison. Key observations include:

• Consistent gains – mean‑preserving MGO‑SC improves test accuracy over AdamW across all noise levels, with the most pronounced benefit at high symmetric noise (e.g., a modest increase of around one percentage point at 40% noise).
• Statistical rigor – results are reported with matched‑seed effect sizes, non‑parametric bootstrap confidence intervals, and paired significance tests, reinforcing the reliability of the observed improvements.
• No extra overhead – the gate adds negligible computational cost and does not require second‑order information, per‑sample gradient filtering, or additional hyper‑parameters beyond the existing AdamW configuration.

The authors also release code and raw training logs, facilitating reproducibility and further exploration by the community.

Practical Takeaways

For practitioners facing noisy training environments—such as datasets with mislabeled examples, small batches, or highly variable data augmentations—MGO‑SC offers a simple plug‑in that can be applied to any Adam‑style optimizer. Implementation requires only a few extra lines to compute the sign‑EMA magnitude and apply the mean‑preserving mask before the standard Adam update. Because the method preserves the overall step magnitude, existing learning‑rate schedules typically remain appropriate, minimizing the need for extensive hyper‑parameter tuning.

Conclusion

MGO‑SC demonstrates that a modest, sign‑consistency‑based gate can meaningfully enhance the robustness of Adam‑style optimizers under noisy gradient conditions. Its ease of integration, low computational footprint, and statistically validated performance gains make it an attractive option for researchers and engineers seeking more reliable training dynamics without sacrificing the simplicity of first‑order methods.