CAFIN: CUDA-accelerated Adaptive Fisher Information Matrix Network Optimization via GPU-Native Kernel Fusion

Introduction

Training deep neural networks with second‑order optimizers can yield faster convergence because they exploit curvature information, but the practical gains are often erased by heavy GPU overhead. AdaFisher addresses this by approximating the Fisher Information Matrix (FIM) with diagonal Kronecker structures and exponential moving averages, yet its reference PyTorch implementation still suffers from a cascade of separate CUDA kernel launches, temporary tensor allocations, and repeated memory traffic. CAFIN (CUDA‑accelerated Adaptive Fisher Information Matrix Network Optimization) tackles the performance bottleneck at the kernel level. By progressively fusing the distinct steps of AdaFisher—FIM construction, EMA updates, gradient preconditioning, momentum handling, weight decay, and parameter updates—into a small set of native CUDA kernels, CAFIN reduces both latency and memory footprint without altering the underlying optimization mathematics.

---

Kernel Fusion Strategy

From Sequential Calls to In‑Place Pipelines

Traditional AdaFisher executes each sub‑operation in its own kernel, forcing the GPU to launch, synchronize, and move intermediate results between global memory and registers. CAFIN rewrites the workflow so that a single kernel can:

1. Load the current gradient and model parameters into registers.
2. Update the EMA of the FIM diagonal blocks directly in shared memory.
3. Apply the preconditioner to the gradient on‑the‑fly.
4. Incorporate momentum and weight‑decay terms.
5. Write the final updated parameters back to global memory.

By keeping data resident in registers or shared memory throughout the step, CAFIN eliminates the need for separate allocation buffers and reduces global‑memory traffic.

Memory‑Efficient Design

• Shared‑memory reuse: The same shared‑memory buffers store both the EMA of the FIM and the intermediate preconditioned gradients, avoiding duplicate allocations.
• Register‑resident computation: Critical scalar operations (e.g., EMA decay, momentum scaling) are performed in registers, which are the fastest on‑chip storage.
• In‑place updates: Parameter tensors are overwritten directly, cutting the per‑step allocation overhead by roughly 19 MB on the tested GPUs.

---

Empirical Highlights

Experiments were conducted on ResNet‑18 across three image classification benchmarks (CIFAR‑10, CIFAR‑100, Tiny ImageNet) using two consumer‑grade GPUs (GTX 1660 Ti and RTX 3060 Laptop). The key observations include:

• FIM construction latency dropped by up to 78 % compared with the baseline AdaFisher implementation.
• Overall optimizer‑step latency improved between 2 % and 27 %, depending on dataset size and GPU.
• Training accuracy remained statistically indistinguishable from the original method across multiple random seeds, confirming numerical fidelity within standard FP32 tolerances.

These results demonstrate that kernel‑level fusion can deliver substantial speedups for second‑order methods without sacrificing the theoretical benefits they provide.

---

Practical Takeaways

1. Performance‑first refactoring: When a high‑level optimizer incurs many small kernels, consider merging them into a single, well‑structured CUDA kernel to minimize launch overhead.
2. Leverage on‑chip memory: Shared memory and registers are underutilized in many deep‑learning pipelines; re‑architecting data flow to keep intermediate values on‑chip can yield both speed and memory savings.
3. Maintain numerical equivalence: CAFIN validates that aggressive fusion does not have to compromise precision; careful ordering of operations and adherence to FP32 rounding behavior preserve the original optimizer’s convergence properties.
4. Portability: Although CAFIN is built for CUDA, the same principles—kernel fusion, in‑place updates, and on‑chip memory reuse—can be adapted to other GPU programming models (e.g., ROCm, SYCL) for broader hardware support.

---

Conclusion

CAFIN showcases how a disciplined, kernel‑fusion approach can revitalize second‑order optimization in deep learning. By collapsing the multiple stages of AdaFisher into a handful of GPU‑native kernels, it slashes both execution time and memory consumption while delivering identical training outcomes. For practitioners seeking to harness curvature‑based methods on commodity hardware, CAFIN provides a concrete blueprint: prioritize kernel consolidation, exploit fast on‑chip storage, and verify numerical consistency throughout the transformation.