Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters
The SageMaker HyperPod Global Inference Gateway lets you use a single, secure endpoint to automatically route AI inference requests across multiple clusters and regions, improving GPU utilization, failover, and operational simplicity.

Introducing the SageMaker HyperPod Global Inference Gateway
Modern AI workloads often span many large models and require massive GPU capacity. Organizations typically spread that capacity across several SageMaker HyperPod clusters and even across AWS Regions. Until now, each cluster required its own endpoint, forcing developers to manage loadâbalancing, DNS, and manual failover. The Global Inference Gateway removes that complexity by providing one SageMakerâmanaged endpoint that intelligently routes requests to the bestâsuited cluster, whether it lives in the same Region or across the backbone to another Region.
---
Why a Global Gateway Matters
| Challenge | Traditional Approach | Global Inference Gateway Solution | |-----------|----------------------|-----------------------------------| | Idle GPU capacity | Requests queue on a saturated cluster while other clusters sit idle. | Realâtime metrics (KV cache, queue depth, error rate, proximity) guide traffic to underâutilized clusters, maximizing GPU usage. | | Manual failover | Engineers must reconfigure endpoints when a cluster fails. | Automatic health checks move traffic away from unhealthy clusters without client changes. | | Operational overhead | Teams build and maintain load balancers, DNS records, and crossâRegion networking. | SageMaker provisions a private network load balancer, VPC endpoint service, and TLS certificates for each clusterâno custom plumbing required. | | Security and compliance | Custom solutions often miss consistent authentication and encryption. | Every request is signed with SigV4, authorized via IAM, and carried over AWS PrivateLink, keeping traffic private by default. |
By consolidating endpoints, the gateway reduces the number of clientâside URLs from modelsâŻĂâŻclusters to one per gateway, simplifying SDK integration and CI/CD pipelines.
---
How to Set Up the Gateway
- Publish clusters â In each HyperPod clusterâs `InferenceGatewayConfig`, enable the `spec.globalRouting` block. SageMaker creates a Network Load Balancer, a VPC endpoint service, and uploads the TLS certificate to your bucket. This step is performed once per cluster.
- Create the gateway â Call `CreateGlobalInferenceGateway` with the list of published clusters you want to include. SageMaker returns a single regional endpoint (e.g., `https://gateway-xyz.region.amazonaws.com`).
- Configure routing weights (optional) â The gateway scores clusters on four signals:
KVHeadroom â Available KV cache and GPU memory (default weightâŻ0.20) QueueDepth â Current request backlog (default weightâŻ0.25) RegionProximity â Network distance to the gatewayâs Region (default weightâŻ0.40) ErrorRate â Recent failure rate (default weightâŻ0.15) Adjust the weights (values 0.0â1.0 that sum to 1.0) to prioritize latency, cost, or capacity as needed.
If a cluster is listed before it is published, the API returns a clear error indicating the missing publication step.
---
Request Flow Overview
- Client â Gateway â The client signs the request with SigV4 and sends it to the gatewayâs regional endpoint.
- Authentication & Scoring â The gateway validates the signature, checks IAM permissions, and scores each healthy cluster using the configured weights.
- PrivateLink Transport â The chosen cluster is reached via AWS PrivateLink, ensuring traffic never traverses the public internet.
- ClusterâLevel Routing â Inside the selected HyperPod cluster, the existing Inference Gateway routes the request to the optimal pod, just as in a singleâcluster deployment.
This endâtoâend flow guarantees low latency, secure transport, and automatic load distribution.
---
Operational Benefits
- Higher GPU utilization â Idle GPUs in one Region can absorb overflow from a busy Region, reducing the need to overâprovision capacity.
- Zeroâdowntime failover â Health checks detect failures instantly; traffic is rerouted without client changes.
- Simplified client code â One endpoint means fewer configuration files, environment variables, and secret rotations.
- Builtâin security â SigV4, IAM, and PrivateLink provide encryption and authentication out of the box.
- Reduced operational burden â SageMaker manages TLS termination, certificate validation, and metric collection, freeing teams to focus on model development.
---
Practical Next Steps
- Identify clusters you want to include in a global gateway (e.g., all production HyperPod clusters across USâEastâ1 and EUâCentralâ1).
- Enable global routing in each clusterâs config and verify the Network Load Balancer and VPC endpoint appear in your account.
- Create the gateway via the SageMaker console or SDK, then update your application to point to the new endpoint.
- Monitor the gatewayâs builtâin metrics (request latency, cluster scores, failover events) in CloudWatch to fineâtune routing weights.
- Iterate â As new clusters come online, publish them and add them to the existing gateway; no client changes are required.
By adopting the SageMaker HyperPod Global Inference Gateway, teams can unlock the full potential of their distributed GPU fleet, achieve seamless crossâRegion resilience, and dramatically simplify the inference architecture.
Source
Read original sourceWhy we picked this
Launches global inference gateway for SageMaker, important AI service feature.