A Guth Labs publication ↗
Back to News

GitHub previews repeatable tests for Copilot skills, agents and tools

Three machined metal blocks inside a green glass test chamber.
AI-generated editorial illustration

GitHub opened Copilot Evals in public preview on November 17, giving developers a way to test how changes to skills, custom agents and MCP tools affect their Copilot setup. The release supports Copilot Individual, Business and Enterprise plans.

Users define a realistic task and the behavior they expect, then inspect both the result and the steps taken to reach it. The same test can be rerun after changing a prompt, model, skill or tool, allowing before-and-after comparisons rather than relying on a single successful answer.

Copilot can create and run tests from natural-language requests or the /eval command. Each run requires approval before it starts, and failed checks can be investigated before the user decides what to change.

The system uses Vally, an extensible local evaluation tool, as its engine. It is distributed as a VS Code extension and plugins for Copilot CLI and the Copilot app.

The preview brings regression testing closer to everyday agent customization, where a configuration change can improve one task while degrading another.

Sources and citations

Revision history

Published .