METR estimates faster AI progress at Anthropic, with major evidence limits

Anthropic's progress on well-specified AI tasks was running at roughly twice its earlier pace as of October, according to an independent METR assessment published November 5. The estimate is uncertain: METR says a rate nearer 1.3 or 3 times the historical pace would not surprise it. It judges that AI-assisted research likely explains most of the current acceleration. METR assessment
The finding comes with a substantial transparency limitation. METR says Anthropic redacted most of the evidence it considered most informative, leaving the public more dependent on the assessors' judgment than the researchers would prefer. The access provided to evaluators and the evidence available to readers are therefore different things. Disclosure limitations
What was measured
The review ran from August 12 to October 29. An internal benchmark index showed capability gains at an estimated 1.9 times the previous rate, with a 95% confidence interval of 1.5 to 2.8. METR says faster scaling of training compute did not explain that acceleration. Its broad judgment concerns well-specified tasks, with uncertainty about less clearly defined work. Methods and results
That distinction matters for interpretation. A doubling in the rate of progress is a comparison between trends over time. It is not a claim that a model is twice as capable at every task or that every researcher finishes an assignment in half the time.
Reuters' account emphasizes that AI was not yet making highly significant, fully autonomous contributions to the most important stages of model training. It also notes that informal staff productivity polls were considered unreliable evidence. Reuters reporting, republished by AOL
A separate review questions the confidence of earlier reporting
METR also published a review of the automated-research section of Anthropic's August risk report. That document concerns models as of July 15, rather than the October state assessed in the broader study. METR agrees that acute catastrophic risks at that earlier point were low, while arguing that the public report understated acceleration and uncertainty. Review of the August risk report
The disagreement focuses partly on an early-warning threshold associated with a doubling of progress attributable to AI research automation. METR says crossing it by July was plausible, although most relevant researchers at both organizations leaned toward it not having been crossed. The threshold is an alert for further scrutiny, not a finding that catastrophic harm has occurred. Threshold assessment
METR also found the argument against complete substitution for Anthropic's research workforce broadly persuasive, while expressing less confidence than before because some evaluations no longer clearly exposed capability limits. Research-automation assessment
The open question is how assistance compounds
The evidence supports taking AI-assisted development seriously without treating autonomous self-improvement as an established outcome. Human researchers can direct, verify and combine AI contributions while the overall development process accelerates. Whether that process becomes more autonomous, and how quickly, remains a separate empirical question.
For readers tracking frontier-model safety, the report is useful for both its estimate and its account of uncertainty. Future assessments will be stronger if they make assumptions, comparison periods and enough supporting evidence public to let outsiders examine how the conclusions were reached.
Sources and citations
Revision history
Published .