Using NVIDIA Run:ai GPU Fractioning on Amazon EKS

NVIDIA Run:ai GPU Fractioning lets multiple pods share a single GPU on Amazon EKS by enforcing per‑pod memory contracts, enabling higher utilization for notebooks, evaluations, and small inference services.

Using NVIDIA Run:ai GPU Fractioning on Amazon EKS

Introduction

GPU sharing on Kubernetes has long been a challenge. Traditional approaches—NVIDIA MIG, device‑plugin time‑slicing, or Multi‑Process Service (MPS)—address different isolation and utilization scenarios but often leave teams waiting for a whole GPU even when memory is available. NVIDIA Run:ai’s GPU Fractioning Operator extends MPS with Kubernetes‑aware scheduling, per‑pod memory contracts, enforcement, and metrics. This walkthrough shows how to set up a custom Amazon Linux 2023 (AL2023) node image and configure GPU‑memory sharing on an Amazon Elastic Kubernetes Service (EKS) cluster using a single g5.xlarge instance with an A10G Tensor Core GPU.

1. Why Choose GPU Fractioning?

  • Enforced memory limits: Each pod declares a GPU‑memory contract (e.g., 8 GiB). The operator guarantees the contract is not exceeded, preventing one workload from starving another.
  • Works on GPUs without MIG: Fractioning is a software‑only solution, so it can be used on A10G, T4, or other GPUs that lack hardware partitioning.
  • Simple node groups: Fixed‑size GPU node groups (e.g., a single‑GPU g5.xlarge) are sufficient; you don’t need the dynamic resource allocation (DRA) APIs.
  • Ideal for intermittent workloads: Interactive notebooks, short model‑evaluation jobs, and low‑throughput inference services benefit most, as they often have idle periods where memory is under‑utilized.

2. Architecture Overview

The validated environment (US‑East‑2) consists of:

  • EKS control plane (Kubernetes 1.36) running in a managed VPC.
  • System node (m7i.large) hosting the Run:ai operator controllers and KAI Scheduler.
  • GPU worker (g5.xlarge) with an A10G GPU (23 028 MiB). The worker runs a custom AL2023 AMI that includes:

- NVIDIA driver R615 series and NVIDIA Container Toolkit. - `containerd` with NRI (Node Resource Interface) integration. - MPS daemon (`fractiond`) for time‑slicing. - `metricsd` daemon exposing per‑pod GPU‑memory usage to Prometheus.

Two example pods each request `nvidia.com/gpu: 1` with an 8 GiB memory contract. The operator enforces the limit, allowing both pods to run concurrently while each actually allocates ~6 GiB. A third pod requesting 9 GiB is rejected, demonstrating the contract enforcement.

3. Implementation Steps

Prerequisites

  1. AWS account with EKS, CloudFormation, VPC, EC2, EBS, IAM, and Systems Manager permissions.
  2. Quotas for `g5.xlarge` and a temporary `m5.xlarge` for image building.
  3. CLI tools: AWS CLI v2, `eksctl`, `kubectl`, Helm, Packer, Git, `jq`, and `curl`.
  4. Temporary credentials (AWSSESSIONTOKEN) if using assumed roles.

Build the Custom AL2023 AMI

  • Clone the AWS Samples repo containing the Packer template.
  • Update the driver version to R615.71.10 (required by GPU Fractioning Operator v0.1.7).
  • Run Packer to produce an AMI that includes the NVIDIA driver, Container Toolkit, and `fractiond`.

Deploy the EKS Cluster

```bash eksctl create cluster \ --name gpu-fractioning-demo \ --region us-east-2 \ --version 1.36 \ --nodegroup-name system-node \ --node-type m7i.large \ --nodes 1 \ --managed ```

  • Install the Run:ai Helm chart (operator + KAI Scheduler) into the `runai` namespace.
  • Apply the GPU Fractioning Operator CRD and create a `GpuFraction` custom resource that defines the default 8 GiB contract.

Create Fraction‑Enabled Pods

```yaml apiVersion: v1 kind: Pod metadata: name: notebook-a annotations: runai.io/gpu-fraction: "8Gi" spec: containers: - name: notebook image: nvcr.io/nvidia/pytorch:24.01-py3 resources: limits: nvidia.com/gpu: 1 ```

  • Deploy a second pod with the same annotation. Both start on the single GPU.
  • Verify allocation with:

```bash kubectl top pod -n runai kubectl get gpunodemetrics ```

  • Attempt a pod with `runai.io/gpu-fraction: "9Gi"`; the scheduler rejects it, confirming enforcement.

Cleanup

```bash eksctl delete cluster --name gpu-fractioning-demo --region us-east-2 aws ec2 deregister-image --image-id <ami-id> ``` All billable resources (EKS control plane, NAT gateway, EC2 instances, EBS volumes, and AMI snapshots) are removed.

4. Practical Takeaways

  • Memory contracts, not compute guarantees: Fractioning limits GPU memory per pod but does not proportionally allocate CUDA cores. Expect similar compute performance to time‑slicing.
  • Fault isolation is limited: The operator provides memory isolation; it does not protect against CUDA‑level crashes. For stricter isolation, consider MIG on supported GPUs.
  • Cost‑effectiveness: By packing two 8 GiB workloads onto a single A10G, you can double the apparent GPU utilization without additional hardware. Actual cost savings depend on workload characteristics and cluster scaling policies.
  • Scalability: The pattern works best with fixed‑size node groups. If you need dynamic scaling based on demand, explore Kubernetes DRA and the full Run:ai KAI Scheduler stack.

Conclusion

NVIDIA Run:ai GPU Fractioning offers a pragmatic way to increase GPU utilization on Amazon EKS when hardware partitioning isn’t available. By enforcing per‑pod memory contracts and integrating with Kubernetes scheduling, it enables multiple low‑intensity workloads to share a single GPU safely. Follow the steps above to build a disposable test environment, validate memory contracts, and assess whether this approach fits your team’s AI development workflow.

Source

Read original source

Why we picked this

Shows GPU fractioning on EKS, a useful AI compute optimization.

← Back to all articles