Fault tolerant distributed training on Amazon EKS using NVRx
What Happened
The article describes integrating NVIDIA Resiliency Extension (NVRx) into PyTorch Fully Sharded Data Parallel (FSDP) training on Amazon Elastic Kubernetes Service (EKS) to address GPU fault tolerance and checkpointing inefficiencies. It explains three NVRx features: async checkpointing, inâprocess restart, and ft_launcher inâjob restart, which together reduce idle time and recover from soft and hard faults without restarting containers. The EKS cluster uses p5.48xlarge instances with H100 GPUs, 32 EFA adapters, and Amazon FSx for Lustre for shared checkpoint storage, enabling fast recovery and highâbandwidth networking.
Why It Matters
Enterprise architects can reduce training downtime and GPU waste by adopting NVRx, lowering operational costs and improving reliability for largeâscale AI workloads. The architecture demonstrates how managed Kubernetes can be combined with specialized GPU and networking resources to meet stringent performance and resilience requirements. Governance can be simplified by keeping application code unchanged while adding faultâtolerance layers through Python imports.
The Limitation
The solution is demonstrated only on H100 GPUs and 2â8 node EKS clusters; performance and faultâtolerance behavior may differ on other GPU types, node counts, or storage backends.
What You Can Do
Implement NVRx async checkpointing and inâprocess restart in your existing PyTorch FSDP training scripts on an EKS cluster with H100 GPUs and FSx for Lustre for shared checkpoints.