A Guth Labs publication ↗
Back to News

OpenAI’s new lab benchmark shows progress and costly gaps in AI judgment

AI-generated conceptual editorial illustration of laboratory bench and abstract analysis panels suggesting uncertainty, not real experimental data; branded guthlabs.ai.
AI-generated conceptual editorial illustration | guthlabs.ai

OpenAI and Ligo tested agents on real experimental evidence. Higher scores still left basic laboratory mistakes.

OpenAI introduced BioBasicsBench on November 6, an evaluation developed with enzyme-design company Ligo that puts AI agents through 100 routine wet-lab analysis tasks. The questions draw on researchers’ own experimental data, including images and instrument readings. OpenAI

At the highest reasoning setting shared by all seven tested models, GPT-6 Astra earned 66.7% of available credit, compared with 43.8% for GPT-5.6 Sol. Those numbers are average rubric scores, not the share of experiments completed successfully. The comparison covers OpenAI’s models, so it does not establish a lead over other developers. OpenAI

The examples show why a better score needs careful interpretation. Astra caught an antibiotic mismatch on a control plate that an older model accepted. In another task, however, it identified visual patterns correctly and then assigned them the wrong experimental meaning, recommending the wrong colonies. R&D World also highlighted a case where Astra overcounted colonies by about 12%. R&D World

The test gives partial credit for separate elements of an answer. Scientists reviewed each task internally and externally, and each model-and-setting combination received five attempts per task. That makes the evaluation useful for locating weaknesses, while keeping the result distinct from an independent test of a deployed laboratory assistant. OpenAI

For a laboratory considering these tools, the practical question is which errors a scientist would have to catch. A persuasive explanation can increase the review burden if its final recommendation quietly contradicts the evidence. An average score alone cannot tell a team how much supervision a particular workflow needs.

BioBasicsBench offers a concrete set of problems to investigate before handing agents more responsibility. A useful next step would be independent testing on other laboratories’ data, with attention to error severity and how reliably models flag uncertainty. That would help turn benchmark progress into a clearer case for everyday use.

Sources and citations

Revision history

Published .