A short traffic spike can leave agent inference slow long after the spike ends
A new preprint argues that agent-serving systems can remain slow after a temporary traffic surge has passed. In controlled tests, Wen Huang and Jack Liu report that prefix-cache misses created a feedback loop: slower requests kept more sessions alive, their accumulated context exceeded available cache capacity, and more context then had to be recomputed. Paper, submitted November 10
In nine of twelve paired runs, the burst-exposed vLLM system remained degraded while its otherwise matched counterpart stayed healthy. The researchers also examined SGLang and a larger Qwen model. Pausing active sessions against a cache-capacity budget helped restore throughput in their tested collapsed runs. Those results describe their experiments, not a universal engine failure rate. Full study
The limits matter: testing used one GPU type and one replica; synthetic inputs and trace-shaped replay did not execute real agents. There was no production evidence, autoscaling, routing or client-retry evaluation. This is a preprint, and no independent replication was found.
Why it matters
Analysis: A fleet's capacity plan should ask what happens after a burst, not just how many requests complete during it. A plausible internal experiment is to replay representative session lengths, inject a bounded burst, then track recovery in cache-hit rate, active context, waiting work and completion latency.
Request count alone may be a poor proxy when some agents retain much longer conversations than others. If a locally hosted system shows this pattern, blindly raising concurrency could make it worse. The right response would depend on measurement and the serving engine, not on copying the paper's thresholds into production.
Evidence status
Primary abstract and full paper read directly. Claims attributed to authors. No software was installed or benchmarks reproduced. This study does not establish that Guth, vLLM generally, or any hosted provider is currently affected.
Sources and citations
Revision history
Published .