Google releases HelpBench to test AI security advice

Google released HelpBench, a 450-question benchmark for testing AI advice about everyday privacy, safety and security, in a November 17 announcement describing research with University College London.
The benchmark pairs questions with expert-written scoring rubrics. Those rubrics assess whether answers include useful factual guidance, avoid harmful or impractical recommendations, prioritize important information and avoid overwhelming the person seeking help.
The public repository includes the questions, an automatic-rater prompt template and scoring code. It also provides expert-scored examples for checking alternative raters. Topics include account recovery, scams, suspected compromise, privacy settings and online harassment.
Google says an evaluation conducted earlier this year covered 18 models from seven companies. The average score was 82%, but roughly one in ten responses scored below 65%, a range the researchers associate with inaccurate or potentially harmful advice. Those results describe the earlier evaluation, not a fresh test of today's newest models.
The release gives developers a shared way to examine whether security advice is usable as well as factually sound. Its rubrics also make burdensome or poorly prioritized recommendations visible rather than measuring factual coverage alone.
Sources and citations
Revision history
Published .