Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters

The SageMaker HyperPod Global Inference Gateway lets you use a single, secure endpoint to automatically route AI inference requests across multiple clusters and regions, improving GPU utilization, failover, and operational simplicity.

Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters

Introducing the SageMaker HyperPod Global Inference Gateway

Modern AI workloads often span many large models and require massive GPU capacity. Organizations typically spread that capacity across several SageMaker HyperPod clusters and even across AWS Regions. Until now, each cluster required its own endpoint, forcing developers to manage load‑balancing, DNS, and manual failover. The Global Inference Gateway removes that complexity by providing one SageMaker‑managed endpoint that intelligently routes requests to the best‑suited cluster, whether it lives in the same Region or across the backbone to another Region.

---

Why a Global Gateway Matters

| Challenge | Traditional Approach | Global Inference Gateway Solution | |-----------|----------------------|-----------------------------------| | Idle GPU capacity | Requests queue on a saturated cluster while other clusters sit idle. | Real‑time metrics (KV cache, queue depth, error rate, proximity) guide traffic to under‑utilized clusters, maximizing GPU usage. | | Manual failover | Engineers must reconfigure endpoints when a cluster fails. | Automatic health checks move traffic away from unhealthy clusters without client changes. | | Operational overhead | Teams build and maintain load balancers, DNS records, and cross‑Region networking. | SageMaker provisions a private network load balancer, VPC endpoint service, and TLS certificates for each cluster—no custom plumbing required. | | Security and compliance | Custom solutions often miss consistent authentication and encryption. | Every request is signed with SigV4, authorized via IAM, and carried over AWS PrivateLink, keeping traffic private by default. |

By consolidating endpoints, the gateway reduces the number of client‑side URLs from models × clusters to one per gateway, simplifying SDK integration and CI/CD pipelines.

---

How to Set Up the Gateway

  1. Publish clusters – In each HyperPod cluster’s `InferenceGatewayConfig`, enable the `spec.globalRouting` block. SageMaker creates a Network Load Balancer, a VPC endpoint service, and uploads the TLS certificate to your bucket. This step is performed once per cluster.
  2. Create the gateway – Call `CreateGlobalInferenceGateway` with the list of published clusters you want to include. SageMaker returns a single regional endpoint (e.g., `https://gateway-xyz.region.amazonaws.com`).
  3. Configure routing weights (optional) – The gateway scores clusters on four signals:

KVHeadroom – Available KV cache and GPU memory (default weight 0.20) QueueDepth – Current request backlog (default weight 0.25) RegionProximity – Network distance to the gateway’s Region (default weight 0.40) ErrorRate – Recent failure rate (default weight 0.15) Adjust the weights (values 0.0‑1.0 that sum to 1.0) to prioritize latency, cost, or capacity as needed.

If a cluster is listed before it is published, the API returns a clear error indicating the missing publication step.

---

Request Flow Overview

  1. Client → Gateway – The client signs the request with SigV4 and sends it to the gateway’s regional endpoint.
  2. Authentication & Scoring – The gateway validates the signature, checks IAM permissions, and scores each healthy cluster using the configured weights.
  3. PrivateLink Transport – The chosen cluster is reached via AWS PrivateLink, ensuring traffic never traverses the public internet.
  4. Cluster‑Level Routing – Inside the selected HyperPod cluster, the existing Inference Gateway routes the request to the optimal pod, just as in a single‑cluster deployment.

This end‑to‑end flow guarantees low latency, secure transport, and automatic load distribution.

---

Operational Benefits

  • Higher GPU utilization – Idle GPUs in one Region can absorb overflow from a busy Region, reducing the need to over‑provision capacity.
  • Zero‑downtime failover – Health checks detect failures instantly; traffic is rerouted without client changes.
  • Simplified client code – One endpoint means fewer configuration files, environment variables, and secret rotations.
  • Built‑in security – SigV4, IAM, and PrivateLink provide encryption and authentication out of the box.
  • Reduced operational burden – SageMaker manages TLS termination, certificate validation, and metric collection, freeing teams to focus on model development.

---

Practical Next Steps

  1. Identify clusters you want to include in a global gateway (e.g., all production HyperPod clusters across US‑East‑1 and EU‑Central‑1).
  2. Enable global routing in each cluster’s config and verify the Network Load Balancer and VPC endpoint appear in your account.
  3. Create the gateway via the SageMaker console or SDK, then update your application to point to the new endpoint.
  4. Monitor the gateway’s built‑in metrics (request latency, cluster scores, failover events) in CloudWatch to fine‑tune routing weights.
  5. Iterate – As new clusters come online, publish them and add them to the existing gateway; no client changes are required.

By adopting the SageMaker HyperPod Global Inference Gateway, teams can unlock the full potential of their distributed GPU fleet, achieve seamless cross‑Region resilience, and dramatically simplify the inference architecture.

Source

Read original source

Why we picked this

Launches global inference gateway for SageMaker, important AI service feature.

← Back to all articles