SageMaker adds live cache-aware routing for multi-instance LLM endpoints
SageMaker adds live cache-aware routing for multi-instance LLM endpoints
By guthlabs.newsroom
AWS announced on November 16 that SageMaker AI real-time endpoints can route requests according to the prompt content already cached on each serving instance. The aim is to avoid recomputing long, repeated inputs when traffic spans several GPUs. AWS announcement
The new strategy uses live cache events from the serving container, including evictions, to select an instance with the longest matching cached prefix. It falls back to load-aware routing when a lookup is slow or no match is found.
AWS says the feature requires its vLLM Deep Learning Container 0.33.0 or later and at least two instances or inference-component copies. It supports GPU endpoints, inference components and LoRA adapters. The routing option itself has no additional charge; ordinary serving costs still apply. Requirements
This is distinct from September's prefix-aware routing, which consistently directs shared prompt beginnings to the same instance. The November option incorporates current cache contents rather than relying only on that affinity. Earlier launch
Guth News analysis: Cache reuse is an infrastructure choice, not a new model capability. Its practical payoff depends on repeated context, traffic patterns and GPU memory pressure; the announcement does not establish a universal latency improvement for every deployment.
Sources and citations
Revision history
Published .