A Guth Labs publication ↗
Back to News

AWS adds managed Dynamo and cache-aware routing to SageMaker inference

Two green glass server monoliths joined by a flowing gold bridge.
AI-generated editorial illustration

AWS announced managed NVIDIA Dynamo support for SageMaker HyperPod Inference and a separate cache-aware routing option for SageMaker AI real-time endpoints on November 17.

The Dynamo integration supports models spread across multiple machines and configurations that separate prompt processing from token generation. These worker pools can scale independently, with cached context transferred over Elastic Fabric Adapter networking. The managed add-on installs serving and scheduling components, while HyperPod monitors hardware health and replaces failed nodes. Supported engines include vLLM, SGLang and TensorRT-LLM.

For conventional SageMaker real-time endpoints, the new routing strategy uses cache information reported by model containers to choose an instance with reusable prompt context and available capacity. It requires a supported AWS vLLM container and at least two instances or inference-component copies to affect routing.

AWS says this endpoint feature carries no additional charge beyond instance costs. It does not apply the tenant-prefix isolation available with prefix-aware routing, an important distinction for shared services.

Both updates target repeated computation and GPU utilization rather than model intelligence. Actual gains depend on prompt reuse, traffic and deployment configuration.

Sources and citations

Revision history

Published .