A Guth Labs publication ↗
Back to News

MMDM exposes the gap between choosing one answer and choosing every valid action

MMDM exposes the gap between choosing one answer and choosing every valid action

A new benchmark asks a practical question about single-pass decision models: can they identify all valid choices when the options depend on one another? MMDM, submitted to arXiv on November 10, evaluates 28 models across 5,448 questions in 16 categories. Its authors report that the strongest Hard Multi-select Accuracy reaches just 36.81%. Paper and submission history

That number requires care. The benchmark counts a multi-select answer as correct only when every valid option ranks above every invalid one. It is not per-option accuracy, a production failure rate, or a verdict on all AI agents. In one reported comparison, Perplexity Decider-V1.2 scores 78.75% on world-knowledge single-select questions but 47.95% on constraint planning. The study also evaluates probability quality separately from answer selection. Full paper

Guth take: score the workflow, not just the answer

A model can be useful at ranking candidates while remaining unsuitable as the sole authority for a consequential action. The distinction matters when eligibility depends on several conditions, when an earlier approval has expired, or when choosing one tool changes what another may do.

For a deployment, I would build tests around those relationships. Include cases with several valid choices, no eligible action, contradictory evidence and changed permissions. Record whether the system abstains, escalates or proceeds, and measure the cost of each error. A well-calibrated-looking number on a single question is not a substitute for that evaluation.

This is a useful research lens for the decision-model category previously covered in Guth’s Gemini Decision article. It does not establish how Gemini Decision 1 performs: its inclusion among tested models was not verified in this pass. Nor does one benchmark establish that a particular workflow will fail.

Evidence, limitations and history

The arXiv abstract, submission history and HTML paper were directly read. Results are the researchers’ reported measurements, not independently replicated here. This is a preprint; peer-review status was not established. The linked project site could not be opened, and no dataset or model execution was performed.

Cited primary sources: arXiv record and paper v1, both November 10, 2026. No independent corroboration or additional background source is cited. Related coverage: Google Gemini Decision API, stable ID google-gemini-decision-api-2026-11-11.

Sources and citations

Revision history

Published .