Classification with Large-Step Gradient Descent: Unifying Self-Stabilization and Convergence Rates
The paper extends a geometric analysis of large‑step gradient descent to classification losses, revealing how curvature‑driven reference subspaces stabilize dynamics, yield explicit convergence bounds, and expose a transient bias that can affect finite‑time alignment with the max‑margin direction.

Introduction
Training neural networks with gradient descent (GD) often involves step sizes that exceed the classic stability limits derived from smooth‑convex analysis. In practice, these large‑step regimes produce non‑monotonic loss curves, oscillatory trajectories, and yet frequently achieve faster convergence and better generalization. Existing theory has offered three complementary lenses: (1) a self‑stabilization view that explains why GD drifts toward flat regions but does not quantify rates; (2) rigorous convergence‑rate results for specific losses that ignore the underlying stabilization mechanism; and (3) a unifying framework for over‑parameterized least‑squares problems that relies on a minimizer manifold. The latter assumption breaks down for classification tasks, where exponential‑tail losses (logistic, cross‑entropy) have no finite minimizer. The present work bridges this gap by extending the geometric perspective to separable linear classification with exponential‑tail losses.
Curvature‑Defined Reference Subspace
When a loss lacks a finite minimizer, the authors replace the minimizer manifold with a reference subspace defined by the curvature of the loss landscape. This subspace captures directions along which the loss decays most sharply. By decomposing the parameter vector into components parallel and orthogonal to this subspace, the dynamics of GD can be expressed as a coupled system:
- Parallel component – drives progress toward lower loss, effectively reducing the sharpness of the landscape.
- Orthogonal component – experiences oscillations that are gradually damped as the parallel direction flattens the curvature.
Under regularity conditions verified for logistic and multiclass cross‑entropy losses on symmetric data models, the authors prove that this decomposition yields explicit bounds on the iterates themselves, not merely on the loss value. The analysis covers both the unstable regime (step size above the traditional threshold) and the stable regime, revealing a three‑phase damping structure in the latter: an initial rapid decay of orthogonal oscillations, a middle phase where damping slows, and a final asymptotic alignment with the reference subspace.
Convergence Rates and Three‑Phase Damping
The derived bounds show that, despite the lack of a finite optimum, GD with large steps still converges to the max‑margin direction—a well‑known implicit bias for separable classification. However, the convergence is not uniform across all directions:
- Phase 1 – Rapid Parallel Progress: The parallel component quickly reduces the loss, flattening the curvature and initiating damping of orthogonal modes.
- Phase 2 – Oscillation Damping: Orthogonal components decay at a rate proportional to the inverse of the step size and the local curvature, leading to a characteristic oscillatory envelope.
- Phase 3 – Asymptotic Alignment: The iterates settle into the max‑margin direction, with residual orthogonal components diminishing to zero.
These phases provide a concrete timeline for practitioners to anticipate when large‑step GD will transition from aggressive loss reduction to stable alignment.
Transient Implicit Bias and Finite‑Time Effects
A notable insight is the transient implicit bias introduced by large step sizes. While the asymptotic limit remains the max‑margin classifier, the early dynamics can leave persistent orthogonal residuals. In certain data geometries—particularly those with asymmetric feature distributions—these residuals may even be amplified before eventually damping out. This phenomenon can affect finite‑time performance, causing temporary misalignment with the optimal decision boundary and potentially influencing generalization on validation data.
Practical Takeaways
- Step‑size selection: Large steps can be beneficial, but practitioners should be aware of the three‑phase damping behavior and monitor orthogonal residuals, especially in early epochs.
- Model diagnostics: Tracking curvature metrics (e.g., sharpness along the reference subspace) can signal when the system enters the damping phases.
- Algorithm design: Hybrid schedules that start with large steps to exploit rapid parallel progress and then reduce the step size to accelerate orthogonal damping may combine the best of both worlds.
Conclusion
By introducing a curvature‑based reference subspace, this work unifies self‑stabilization and convergence‑rate analyses for large‑step GD in classification settings lacking a finite minimizer. The resulting explicit iterate bounds, three‑phase damping description, and identification of a transient implicit bias deepen our theoretical understanding and offer concrete guidance for leveraging large‑step GD in practice.
Source
Read original sourceWhy we picked this
Theoretical work on gradient descent regimes, significant AI/ML contribution.