Using NVIDIA Run:ai GPU Fractioning on Amazon EKS
NVIDIA Run:ai GPU Fractioning lets multiple pods share a single GPU on Amazon EKS by enforcing perâpod memory contracts, enabling higher utilization for notebooks, evaluations, and small inference services.

Introduction
GPU sharing on Kubernetes has long been a challenge. Traditional approachesâNVIDIA MIG, deviceâplugin timeâslicing, or MultiâProcess Service (MPS)âaddress different isolation and utilization scenarios but often leave teams waiting for a whole GPU even when memory is available. NVIDIA Run:aiâs GPU Fractioning Operator extends MPS with Kubernetesâaware scheduling, perâpod memory contracts, enforcement, and metrics. This walkthrough shows how to set up a custom Amazon Linux 2023 (AL2023) node image and configure GPUâmemory sharing on an Amazon Elastic Kubernetes Service (EKS) cluster using a single g5.xlarge instance with an A10G Tensor Core GPU.
1. Why Choose GPU Fractioning?
- Enforced memory limits: Each pod declares a GPUâmemory contract (e.g., 8âŻGiB). The operator guarantees the contract is not exceeded, preventing one workload from starving another.
- Works on GPUs without MIG: Fractioning is a softwareâonly solution, so it can be used on A10G, T4, or other GPUs that lack hardware partitioning.
- Simple node groups: Fixedâsize GPU node groups (e.g., a singleâGPU g5.xlarge) are sufficient; you donât need the dynamic resource allocation (DRA) APIs.
- Ideal for intermittent workloads: Interactive notebooks, short modelâevaluation jobs, and lowâthroughput inference services benefit most, as they often have idle periods where memory is underâutilized.
2. Architecture Overview
The validated environment (USâEastâ2) consists of:
- EKS control plane (Kubernetes 1.36) running in a managed VPC.
- System node (m7i.large) hosting the Run:ai operator controllers and KAI Scheduler.
- GPU worker (g5.xlarge) with an A10G GPU (23âŻ028âŻMiB). The worker runs a custom AL2023 AMI that includes:
- NVIDIA driver R615 series and NVIDIA Container Toolkit. - `containerd` with NRI (Node Resource Interface) integration. - MPS daemon (`fractiond`) for timeâslicing. - `metricsd` daemon exposing perâpod GPUâmemory usage to Prometheus.
Two example pods each request `nvidia.com/gpu: 1` with an 8âŻGiB memory contract. The operator enforces the limit, allowing both pods to run concurrently while each actually allocates ~6âŻGiB. A third pod requesting 9âŻGiB is rejected, demonstrating the contract enforcement.
3. Implementation Steps
Prerequisites
- AWS account with EKS, CloudFormation, VPC, EC2, EBS, IAM, and Systems Manager permissions.
- Quotas for `g5.xlarge` and a temporary `m5.xlarge` for image building.
- CLI tools: AWS CLI v2, `eksctl`, `kubectl`, Helm, Packer, Git, `jq`, and `curl`.
- Temporary credentials (AWSSESSIONTOKEN) if using assumed roles.
Build the Custom AL2023 AMI
- Clone the AWS Samples repo containing the Packer template.
- Update the driver version to R615.71.10 (required by GPU Fractioning Operator v0.1.7).
- Run Packer to produce an AMI that includes the NVIDIA driver, Container Toolkit, and `fractiond`.
Deploy the EKS Cluster
```bash eksctl create cluster \ --name gpu-fractioning-demo \ --region us-east-2 \ --version 1.36 \ --nodegroup-name system-node \ --node-type m7i.large \ --nodes 1 \ --managed ```
- Install the Run:ai Helm chart (operator + KAI Scheduler) into the `runai` namespace.
- Apply the GPU Fractioning Operator CRD and create a `GpuFraction` custom resource that defines the default 8âŻGiB contract.
Create FractionâEnabled Pods
```yaml apiVersion: v1 kind: Pod metadata: name: notebook-a annotations: runai.io/gpu-fraction: "8Gi" spec: containers: - name: notebook image: nvcr.io/nvidia/pytorch:24.01-py3 resources: limits: nvidia.com/gpu: 1 ```
- Deploy a second pod with the same annotation. Both start on the single GPU.
- Verify allocation with:
```bash kubectl top pod -n runai kubectl get gpunodemetrics ```
- Attempt a pod with `runai.io/gpu-fraction: "9Gi"`; the scheduler rejects it, confirming enforcement.
Cleanup
```bash eksctl delete cluster --name gpu-fractioning-demo --region us-east-2 aws ec2 deregister-image --image-id <ami-id> ``` All billable resources (EKS control plane, NAT gateway, EC2 instances, EBS volumes, and AMI snapshots) are removed.
4. Practical Takeaways
- Memory contracts, not compute guarantees: Fractioning limits GPU memory per pod but does not proportionally allocate CUDA cores. Expect similar compute performance to timeâslicing.
- Fault isolation is limited: The operator provides memory isolation; it does not protect against CUDAâlevel crashes. For stricter isolation, consider MIG on supported GPUs.
- Costâeffectiveness: By packing two 8âŻGiB workloads onto a single A10G, you can double the apparent GPU utilization without additional hardware. Actual cost savings depend on workload characteristics and cluster scaling policies.
- Scalability: The pattern works best with fixedâsize node groups. If you need dynamic scaling based on demand, explore Kubernetes DRA and the full Run:ai KAI Scheduler stack.
Conclusion
NVIDIA Run:ai GPU Fractioning offers a pragmatic way to increase GPU utilization on Amazon EKS when hardware partitioning isnât available. By enforcing perâpod memory contracts and integrating with Kubernetes scheduling, it enables multiple lowâintensity workloads to share a single GPU safely. Follow the steps above to build a disposable test environment, validate memory contracts, and assess whether this approach fits your teamâs AI development workflow.
Source
Read original sourceWhy we picked this
Shows GPU fractioning on EKS, a useful AI compute optimization.