A Guth Labs publication ↗
Back to News

Ragas 0.4.4 treats the evaluator itself as an attack surface

Ragas 0.4.4 treats the evaluator itself as an attack surface

Ragas released version 0.4.4 on November 14, adding a security fix to the software many teams use to evaluate retrieval-augmented AI. PyPI's release record confirms the package and date.

The maintainers' release notes say the patch addresses CVE-2026-6587 across multimodal faithfulness and relevance metrics. Untrusted image references in evaluation inputs could otherwise trigger internal-network requests or local-image reads whose results reach a model provider. Remote images are restricted to public addresses; local files now require explicit permission and a base directory. Blocked images become text with a warning, potentially changing scores. The release also repairs dependency imports and Anthropic compatibility.

A new patch, not a newly disclosed vulnerability

The GitHub-reviewed advisory was published April 20 and updated April 22, 2026. It lists affected versions 0.2.3 through 0.4.3 and still says no patched version. That stale field conflicts with the November release claim, not with the existence of a release. The advisory rates the issue low severity. This article does not upgrade that rating or imply active exploitation.

Guth analysis

Evaluation infrastructure deserves a threat model of its own. A dataset is not necessarily passive when a metric follows embedded image references. Teams often focus on the production agent's permissions while giving evaluation jobs broad access to development machines or shared networks. The useful question is not merely whether a benchmark passes, but what its evaluator is allowed to retrieve while producing that score.

For affected installations, a sensible review combines the patch with restricted network and filesystem access. Regression testing should check both safety boundaries and changed scoring behavior; restoring unrestricted access merely to reproduce an old score defeats the reason for the change.

Evidence limits and revision history

This is a maintainer-reported fix, not an independently reproduced security assessment. No deployment was tested, no new exploit is asserted, and no independent reporting confirming patch efficacy was found. GitHub's release date and PyPI agree on November 14; exact release timezone was not verified. The April advisory is historical context, not November breaking news.

Sources and citations

Revision history

Published .