AWS adds one SageMaker HyperPod inference endpoint across clusters

Amazon Web Services introduced SageMaker HyperPod Global Inference Gateway on November 20. The managed service gives an application one endpoint for models running on registered HyperPod clusters, even when those clusters are in different AWS regions.
The gateway checks cluster health and load, then chooses where to send a request. AWS says it automatically redirects traffic when a cluster fails and carries requests to the cluster over its private network. Customers choose which clusters to register and can adjust how the gateway weighs routing signals. The gateway chooses the cluster, while the existing local routing layer still chooses the server that runs the model within that cluster.
AWS introduced the per-cluster gateway in September and listed cross-cluster routing as a future feature. The new layer matters for teams running models in several places: they can use available capacity in another cluster without making every application track individual endpoints or reroute traffic itself.
Sources and citations
Revision history
Published .