A Guth Labs publication ↗
Back to News

AI vision needs dedicated capacity, not necessarily a separately pretrained encoder

A new multimodal model is usually introduced as a package deal. A November 12 research paper from Meta FAIR and NYU asks a more useful engineering question: which parts of the visual processing recipe actually earn their cost?

The authors separate three choices: learning visual features beforehand, reserving parameters specifically for images, and placing those parameters before images meet language. Their controlled experiments find that an encoder trained jointly from scratch can match a separately pretrained one. Simply deleting the visual capacity does not work as well: enlarging the shared language backbone by the same parameter budget does not close the visual-understanding gap. Paper and submission history

Move the capacity, rather than remove it

The authors’ paper reports that image-only modules in early backbone layers can perform comparably to a separate encoder. Its serving test uses a 1.7B backbone on one H200, with 16 concurrent requests containing an image and text. That is a specific laboratory workload, not a promise about a laptop or every deployed vision service. Full paper

The limitations are central. The experiments use dense backbones up to 6.4B parameters. Encoders are trained on common images to make the comparison fair, so the result does not rule out a benefit from an existing encoder trained on additional data. Higher-resolution training, sparse experts and different image-text mixtures remain open questions. Full paper, discussion

Guth analysis

The takeaway is not “throw away vision encoders.” It is that architecture labels can hide several independent decisions. For an open-model team, changing where image-specific capacity sits may be a more promising experiment than moving those parameters into a larger general-purpose backbone.

This is useful research rather than a new downloadable frontier-model launch. Its strongest contribution is the controlled comparison: it gives builders a reason to test individual design choices instead of copying an entire fashionable recipe. Independent reproduction was not located, and these author-reported results should not be promoted to a universal model-design rule.

Event date: November 12, 2026. Newly discovered coverage on November 15, not breaking news.

Sources and citations

Revision history

Published .