A Guth Labs publication ↗
Back to News

Deepnote tests whether proactive agents know when not to help

Deepnote tests whether proactive agents know when not to help

Deepnote's Agent Initiative Benchmark tests 12 models across 50 workplace situations. Its useful distinction is that an accurate answer can still be an unwanted interruption. The creator reports that models stayed silent in roughly six out of ten situations requiring restraint.

The test uses a synthetic company and a 180-second model-response budget, excluding tool execution. GPT-6 Astra scores 8.97/10 and GPT-6.1 Sol 8.65, according to Deepnote. These are results under a fixed setup, not universal rankings. Two model judges score contributions, and the test does not measure a team's longer-term response.

The repository supplies eight runnable examples and simulated service tools without requiring live company accounts. Code and tasks use Apache 2.0. The underlying PageMantle dataset is separate and uses CC BY 4.0, according to the repository. Its precise release date was not reverified for publication.

Guth analysis: Proactive-agent evaluation should penalize unnecessary contact independently of factual correctness. A polished message about a resolved issue still costs the recipient attention. Testing silence alongside useful intervention is a better fit for always-on assistants than measuring answer quality alone.

Evidence limits: No independent reproduction was found or performed. The primary benchmark page is undated; November 13 discovery reports are not treated as proof of its precise launch time. A fixed snapshot and one effort setting constrain generalization.

Sources and citations

Revision history

Published .