Fault tolerant distributed training on Amazon EKS using NVRx
What Changed
The article describes integrating NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (EKS) to address GPU fault tolerance and checkpointing inefficiencies. It explains three NVRx features: async checkpointing, in‑process restart, and ft_launcher in‑job restart, which together reduce idle time and recover from soft and hard faults without restarting containers. The EKS cluster uses p5.48xlarge instances with H100 GPUs, 32 EFA adapters, and Amazon FSx for Lustre for shared checkpoint storage, enabling fast recovery and high‑bandwidth networking.
Why It Matters
Enterprise architects can reduce training downtime and GPU waste by adopting NVRx, lowering operational costs and improving reliability for large‑scale AI workloads. The architecture demonstrates how managed Kubernetes can be combined with specialized GPU and networking resources to meet stringent performance and resilience requirements. Governance can be simplified by keeping application code unchanged while adding fault‑tolerance layers through Python imports.
The Limitation
The solution is demonstrated only on H100 GPUs and 2‑8 node EKS clusters; performance and fault‑tolerance behavior may differ on other GPU types, node counts, or storage backends.
What You Can Do
Implement NVRx async checkpointing and in‑process restart in your existing PyTorch FSDP training scripts on an EKS cluster with H100 GPUs and FSx for Lustre for shared checkpoints.