A Guth Labs publication ↗
Back to News

Two local-model studies show why disagreement is not the same as useful uncertainty

Two local-model studies show why disagreement is not the same as useful uncertainty

Two recent preprints test an appealing idea for local AI: use inexpensive precision changes to decide when an answer deserves more computation. The results are more qualified than a simple “second opinion” pitch.

In One Model, Two Precisions, publicly submitted November 12, researchers test simulated quantized copies of Qwen3-0.6B and 1.7B on 8,398 multiple-choice questions. A lower-precision disagreement score barely improves on the answering model’s existing entropy signal alone. Adding disagreement to confidence features gives a small gain on 0.6B, while the selected 1.7B result does not clearly exceed noise. At equal total weight traffic, routing using the free entropy signal beats the extra-pass approaches. These are author-reported results, not independent reproduction. Study · Full paper

The related November 10 Precision on Demand study retries uncertain decoding steps at eight bits after an initial four-bit pass. On 150 GSM8K problems with Qwen3-0.6B, one gate scores 62.7% against 64.0% for all-high precision while using about 87% of modeled weight bytes. Wide intervals prevent a firm accuracy conclusion. This is simulated quantization and a byte-traffic calculation, not demonstrated wall-clock or battery savings. Study · Full paper

Guth analysis: A routing signal should earn its own inference budget. Comparing only detection accuracy can hide the cost of producing the score. Test the complete route, including fallback work, and distinguish a changed answer from a corrected one.

Dates and limits: Catch-up research coverage, not breaking. The disagreement PDF is dated November 10, but arXiv records its public submission on November 12. Neither paper establishes reliability for general tool use or authorization decisions.

Sources and citations

Revision history

Published .