A Guth Labs publication ↗
Back to News

METR estimates faster AI progress at Anthropic, with major evidence limits

AI-generated conceptual laboratory illustration with two upward light trails and partly obscured measurements, representing research acceleration and evidence limits.
AI-generated editorial illustration by guthlabs.newsroom

Anthropic's progress on well-specified AI tasks was running at roughly twice its earlier pace as of October, according to an independent METR assessment published November 5. The estimate is uncertain: METR says a rate nearer 1.3 or 3 times the historical pace would not surprise it. It judges that AI-assisted research likely explains most of the current acceleration. METR assessment

The finding comes with a substantial transparency limitation. METR says Anthropic redacted most of the evidence it considered most informative, leaving the public more dependent on the assessors' judgment than the researchers would prefer. The access provided to evaluators and the evidence available to readers are therefore different things. Disclosure limitations

What was measured

The review ran from August 12 to October 29. An internal benchmark index showed capability gains at an estimated 1.9 times the previous rate, with a 95% confidence interval of 1.5 to 2.8. METR says faster scaling of training compute did not explain that acceleration. Its broad judgment concerns well-specified tasks, with uncertainty about less clearly defined work. Methods and results

That distinction matters for interpretation. A doubling in the rate of progress is a comparison between trends over time. It is not a claim that a model is twice as capable at every task or that every researcher finishes an assignment in half the time.

Reuters' account emphasizes that AI was not yet making highly significant, fully autonomous contributions to the most important stages of model training. It also notes that informal staff productivity polls were considered unreliable evidence. Reuters reporting, republished by AOL

A separate review questions the confidence of earlier reporting

METR also published a review of the automated-research section of Anthropic's August risk report. That document concerns models as of July 15, rather than the October state assessed in the broader study. METR agrees that acute catastrophic risks at that earlier point were low, while arguing that the public report understated acceleration and uncertainty. Review of the August risk report

The disagreement focuses partly on an early-warning threshold associated with a doubling of progress attributable to AI research automation. METR says crossing it by July was plausible, although most relevant researchers at both organizations leaned toward it not having been crossed. The threshold is an alert for further scrutiny, not a finding that catastrophic harm has occurred. Threshold assessment

METR also found the argument against complete substitution for Anthropic's research workforce broadly persuasive, while expressing less confidence than before because some evaluations no longer clearly exposed capability limits. Research-automation assessment

The open question is how assistance compounds

The evidence supports taking AI-assisted development seriously without treating autonomous self-improvement as an established outcome. Human researchers can direct, verify and combine AI contributions while the overall development process accelerates. Whether that process becomes more autonomous, and how quickly, remains a separate empirical question.

For readers tracking frontier-model safety, the report is useful for both its estimate and its account of uncertainty. Future assessments will be stronger if they make assumptions, comparison periods and enough supporting evidence public to let outsiders examine how the conclusions were reached.

Sources and citations

Revision history

Published .