Infrathon shows why a short agent benchmark can pick the wrong long-running worker
Infrathon shows why a short agent benchmark can pick the wrong long-running worker
By guthlabs.newsroom
A November 9 preprint argues that the time allowed for an agent to work can change which model looks strongest. Infrathon studies infrastructure optimization across 25 tasks, six models and 150 runs. The authors report that 135 runs improved their best result during the latter half of their allotted time, and that relative rankings shifted with the evaluation horizon. Submission record; full paper
The project covers GPU kernels, inference and training systems, with task budgets of 12 to 24 hours. Its repository describes a separate judging environment that checks submissions against held-out workloads, alongside an integrity review for rule violations. The score reflects the best valid submission within the budget, not the agent's claim that it finished. Project repository
The authors also identify unreliable testing, task violations and early stopping as obstacles. Those limitations are central to the result: more time does not eliminate the need to check what an agent changed. Behavior analysis
Guth analysis: A five-minute leaderboard should not automatically choose the worker for an overnight optimization job. The evaluation needs to resemble the actual assignment, including its time budget, hardware, correctness constraints and recovery requirements. Conversely, a successful long run is not evidence that every job benefits from a larger budget.
For a governed fleet, the useful pattern is independent evidence of valid progress. Keep the agent's work log separate from the measurement that accepts its result; preserve failed attempts rather than reporting only the best score.
Evidence limits: This is newly discovered November 9 research, not breaking news. Results are author-reported and have not been reproduced here. The paper and current repository give different headline compression ratios; this draft intentionally omits that number pending reconciliation.
Sources and citations
Revision history
Published .