AWS adds managed Dynamo and cache-aware routing to SageMaker inference

AWS announced managed NVIDIA Dynamo support for SageMaker HyperPod Inference and a separate cache-aware routing option for SageMaker AI real-time endpoints on November 17.
The Dynamo integration supports models spread across multiple machines and configurations that separate prompt processing from token generation. These worker pools can scale independently, with cached context transferred over Elastic Fabric Adapter networking. The managed add-on installs serving and scheduling components, while HyperPod monitors hardware health and replaces failed nodes. Supported engines include vLLM, SGLang and TensorRT-LLM.
For conventional SageMaker real-time endpoints, the new routing strategy uses cache information reported by model containers to choose an instance with reusable prompt context and available capacity. It requires a supported AWS vLLM container and at least two instances or inference-component copies to affect routing.
AWS says this endpoint feature carries no additional charge beyond instance costs. It does not apply the tenant-prefix isolation available with prefix-aware routing, an important distinction for shared services.
Both updates target repeated computation and GPU utilization rather than model intelligence. Actual gains depend on prompt reuse, traffic and deployment configuration.
Sources and citations
Revision history
Published .