A Guth Labs publication ↗
Back to News

MKA tests whether AI agents can turn dense visual files into usable professional work

Architectural plans and an engineering drawing connect through a magnifying glass to a report, presentation and spreadsheet.
AI-generated editorial illustration by guthlabs.newsroom

Google DeepMind and Artificial Analysis launched the Multimodal Knowledge Agent benchmark on November 10, testing work that requires more than extracting text from a document. MKA asks agents to interpret visually complex inputs and produce finished artifacts such as reports, spreadsheets and presentations. Google announcement

The evaluation contains 200 tasks spanning 83 occupations and ten sectors. Domain experts wrote the tasks and their grading rubrics. Google says the work would typically take an experienced professional three to eight hours, with rubrics weighted 75% toward factual accuracy and 25% toward visual design. Task design

The headline score is not a completed-job percentage

Artificial Analysis reports a leading Climb Weighted Satisfaction Rate of 57% for Claude Fable 5.2 at maximum effort, followed by Gemini 4 Carbon at high effort on 54% and Claude Opus 5.5 at maximum effort on 53%. These are the published results at launch. Artificial Analysis launch results

Climb criteria are the harder rubric requirements. Regression criteria check baseline competence. The headline metric averages the weighted share of Climb requirements satisfied across tasks; it does not mean an agent independently completed 57% of entire professional assignments. Scoring methodology

That distinction is consequential for buyers. A document can satisfy many individual requirements and still contain a single error that makes it unusable. Teams evaluating high-stakes work need to identify those critical failures separately from an aggregate score.

A common working environment improves comparability

Artificial Analysis runs the agents in its Stirrup harness with code execution, web tools and image viewing, subject to a 250-turn limit. An agentic judge inspects submitted deliverables against the rubric. The main 200-task set stays private, while Google has released ten separate public examples. Evaluation environment, Public and private sets

A common setup reduces one source of variation, but results still describe that setup. An organization with different document converters, permissions or review tools may see different outcomes. The public examples are useful for understanding the task format without treating them as a substitute for the private evaluation.

Cost and specialty can change the choice

Artificial Analysis reports average costs of $31.97 per task for Fable 5.2 and $6.29 for Carbon. It also finds that no model leads all ten sectors. The difference between a top aggregate score and the best fit for a particular department is therefore material. Cost and sector results

A practical internal trial should include the documents people actually struggle with: dense drawings, scanned pages, complicated tables and mismatched references. Grade the resulting files for correctness, completeness and usability, then add review time and repeat attempts to the cost.

MKA offers a more relevant comparison for visually demanding knowledge work. Its strongest use is to help narrow a shortlist and design a better local test, while keeping the final acceptance decision tied to the finished work rather than a single leaderboard rank.

Sources and citations

Revision history

Published .