GitHub Copilot Evals Can Test Your Skills. Here’s a First Case.

A Copilot skill can produce a convincing result while quietly breaking one of your team’s rules. GitHub Copilot Evals gives you a way to turn that rule into a repeatable check. For a first eval, pick one task you already trust yourself to judge, give it fixed inputs, and save a baseline before editing the skill.

Two illustrated paper instruction cards beside a brass ruler and three wooden check marks.
AI-generated concept illustration of comparing two versions against the same checks.

GitHub announced public preview on November 17, 2026 for Copilot Individual, Business and Enterprise plans. The release covers VS Code, Copilot CLI and the Copilot app. This guide uses the desktop VS Code extension and a documentation skill as an example.

Pick a task with a checkable output

Suppose your team has a skill that drafts a support FAQ from a fictional product brief. Give it a short brief with two confirmed facts and one unresolved question. Ask it to produce a Markdown FAQ. A clear eval checks that both confirmed facts appear, the open question is marked as unresolved, and the source brief stays unchanged. Those checks catch concrete failures that a generic instruction such as “make the FAQ helpful” leaves ambiguous.

Keep the brief in a small test project you control. Use fictional content so the example is easy for a teammate to inspect and repeat. If your team is still deciding what belongs in its skill instructions, see our guide to giving a coding agent a browser support target for another example of making a requirement explicit.

Review the eval before its first run

The extension listing documents prerequisites including a trusted local project in desktop VS Code 1.138.0 or later, Copilot sign-in and test runner setup. Open the Evals sidebar, find your skill under Assets and choose the action to create evals with Copilot. Review the proposed task, inputs and checks before saving. If the generated check rewards the right tone but misses an unsupported claim, tighten the criterion before treating the result as useful evidence.

Save a baseline and change one thing

Run the saved eval and review the execution details and estimated credit information shown for approval. Open its result and read the actual FAQ alongside the pass and fail list. Keep that run as your baseline. If the FAQ invents an answer to the open question, make one focused edit to the skill that tells it how to handle missing information. Rerun with the same sample, criteria and model where practical. The extension can compare saved runs.

Check whether the improvement holds

Read both outputs. A new pass matters only if the confirmed facts still survive and the unresolved question stays visible. If you plan to rely on the skill across several projects, add a second input with a different omission before you adopt the change. AI outputs can vary across runs, so treat an eval as evidence about the cases you tried. Save the cases that represent failures your team would care about next time.

Prepared with AI assistance from GitHub’s announcement and extension documentation. The FAQ case is illustrative and was not run by TechPulse. See our editorial policy.

MD Rafikul Islam

Written by

MD Rafikul Islam is a software developer and editor of TechPulse. He writes about developer tools, hardware, AI, and practical technology decisions. Some articles are based on cited documentation and analysis rather than hands-on testing; readers should check each article for sources and testing disclosures. Corrections are welcome at rony.yf25@gmail.com.

✍️ Leave a Comment

Your email address will not be published. Required fields are marked *