A Guth Labs publication ↗
Back to News

Redwood tests whether simulated safety checks can stand in for real computers

A research report offers a limited but useful result for safer AI evaluations: monitors reading a model’s internal activations caught attacks at similar rates in simulated and real computer environments. Researchers tested six probes against Qwen3.5-27B on 1,313 LinuxArena tasks. Average detection in simulation was one percentage point lower, with a 95% interval from minus three to plus one. Research repository

That finding does not establish universal equivalence. The study covers one agent, one simulator and one environment suite. Its 1% false-positive setting is per command or tool-output unit, not per full run. A run counts as caught if any unit crosses the threshold. Research repository

The public repository includes scores, analysis and probe weights. It does not include the agent run logs, activations or the separate private simulator, limiting end-to-end reproduction. Publication of supporting artifacts is valuable, but should not be described as a fully self-contained reproduction package.

Guth analysis: Simulation is attractive because evaluators should not need to expose production systems to establish that a monitor works. This study supports further testing of that approach, not blanket confidence. The more important next result is whether the agreement survives stronger agents that recognize their test environment and change behavior.

Sources and citations

Revision history

Published .