Skip to content

Testing and Evals

Every skill in this repo ships with a mirrored test package so reviewers (and future automation) can judge it consistently.

The test package

tests/<category-folder>/<skill-folder>/
├── prompt.md      # Required: a locked, realistic reviewer prompt for the skill
├── rubric.md      # Required: pass conditions, scored criteria, failure conditions, threshold
└── expected.md    # Optional: exemplar output or must-hit expectations for narrow tasks
  • prompt.md should be concrete, realistic, and self-contained; it's the standard scenario every reviewer runs
  • rubric.md is the primary review standard for open-ended skills (writing, review, design)
  • expected.md suits narrow tasks with must-hit findings; it should never be the only mechanism for open-ended skills

Start from tests/templates/ when creating a new package. CI requires prompt.md and rubric.md to exist for every skill.

The two-agent eval workflow

For manual evals, use the workflow in tests/EVAL_FRAMEWORK.md. It separates generation from grading so neither agent biases the other:

  1. Agent 1 (skill-user): reads the skill and the locked prompt.md, produces only the candidate output
  2. Agent 2 (skill-evaluator): reads only the candidate output, scores it against rubric.md

Report results in your PR: prompt path, rubric path, candidate output summary, and pass/fail.

When tests must change

If skill behavior changes, update the test package in the same PR. CI only checks that the files exist; alignment between the skill and its tests is a human reviewer's job.

Quality bar (from CONTRIBUTING)

A good skill is:

  1. Valuable: enhances or automates real consulting/day-to-day tasks
  2. Proven: multiple examples of success, consistent output
  3. Specific: minimizes divergence between intent and outcome
  4. Concise: preserves context window; references carry only what's needed

Validate locally before opening a PR:

uvx --from skills-ref agentskills validate path/to/skill_folder