A Guth Labs publication ↗
Back to News

TCSExam puts machine-checkable results ahead of persuasive research prose

TCSExam puts machine-checkable results ahead of persuasive research prose

A November 5 preprint introduces TCSExam, a research benchmark spanning theorem proving, solver engineering, constructions and open conjectures. Full contains 100 expert-reviewed tasks, while Lite uses 20. Submissions face Lean-kernel verification or executable checking rather than only a model judge. This is newly discovered coverage of older research, not a November 13 launch. Paper

The authors report GPT-6 Astra leading Full and Opus 5.5 leading Lite. Those are results for configured model-agent systems, not proof that either model can independently settle arbitrary open problems. The abstract describes verified special cases and bounds as well as other research outputs. Preprint

The official documentation makes an important distinction: Full defaults to online solver feedback, while Lite defaults to offline judging after development. Both protocols are supported, but should be reported separately. The Index is a composite in points and can exceed 100; it is not a simple percentage of solved tasks. Code and checkers use Apache-2.0, while documentation, paper and third-party materials have separate terms. Project documentation

Guth analysis

A verifier changes what “done” can mean. Instead of trusting an agent's confident summary, an evaluator can demand a proof object or a program that passes a specified check. Yet verification is always relative to the statement and checker: a valid artifact for the wrong problem is not a useful outcome.

The transferable lesson for agent assurance is to retain the task definition, checker version, execution budget and raw output together. Do not flatten a failed judging run into an ordinary zero score or compare online-feedback and offline protocols as though they offered the same information.

Evidence limits

The benchmark was not executed for this report. Results remain author-reported; no independent replication was established. Formal checking of an artifact should not be confused with broad reliability outside the benchmark.

Sources and citations

Revision history

Published .