Skill Eval Harness

Skill Eval Harness is a Python CLI that measures the causal lift of an Agent Skill: it runs the same case, model, and repetition with and without the skill, validates that exact experimental identity, then reports what changed, what passed, and whether the eval leaked its own answer. It reads evals/shared-benchmark.json, emits answer-key-safe task rows, grades files under eval-runs/ locally and deterministically — no model call in the grade path — and writes benchmark reports you can diff across variants.

General eval frameworks (openai/evals, vitest-evals, viteval) score one output against a rubric. This one measures the difference the skill makes, and spends its surface area on keeping that difference honest: paired with/without comparison, tune/holdout/holdback split discipline, leakage lint, materialized ablations with provenance gates, and per-model lift. None of those frameworks have them, and they are what make a reported number trustworthy rather than merely green.

Questions this helps answer

Question	Command/report to use
Does this skill improve outputs compared with no skill at all?	prepare paired with_skill / without_skill rows, then benchmark paired lift and significance.
Which prompts improved, regressed, saturated, or showed no lift?	benchmark case_flags, render-viewer, and error-analysis.
Is the skill worth its extra tokens or dollars?	profile-skill, token-overhead, cost-summary, and lift-per-dollar summaries.
Did my latest skill edit introduce a regression?	Re-run the same manifest, inspect ablation_regressions, trend, and render-viewer --previous-workspace.
Which instruction, checklist, reference, script, or asset is load-bearing?	Materialized ablation:<id> arms plus declared expected_regressions.
Does the agent discover/load the skill when it should, and stay quiet when it should not?	skill-trigger-matrix or skill-pi-trigger-eval, split by should-fire / should-not-fire cases.
Which model tier should this skill target?	prepare --models, then benchmark by_model and model_analysis.
Is this eval safe to spend model budget on?	validate --strict-leakage --leakage-min-chars 1 --check-ablations and audit-manifest --fail-on-blockers.
Can I trust this LLM judge or rubric result?	judge, compare-judges, judge-robustness, and judge-alignment.
Could the eval be contaminated by leaked answer keys or memorized canaries?	Prompt leakage lint plus contamination over generated outputs.
Can this become a CI gate?	suite-run, `report --format junit

Core loop

1. Describe cases in evals/shared-benchmark.json: prompt, split, fixture files, variants, assertions, and ablations.
2. Prepare tasks with skill-benchmark prepare; generation rows omit expected_behavior and judge rubrics unless you explicitly request them.
3. Run tasks with Claude, Codex, Gemini CLI, Mistral Vibe, Jetty, or any runner that writes the run-output contract; Pi support is currently trigger-focused plus workspace-specific smoke tooling.
4. Grade outputs with deterministic assertions: string, regex, file, JSON field, and opt-in script oracles.
5. Inspect the report for pass rates, flaky repeated runs, no-lift cases, saturated assertions, judge tasks, and trigger/no-trigger results.

What the CLI owns

• Causal lift: exact (case, model, repetition, population) with_skill vs without_skill pairs (plus optional old_skill and ablation:<id>), with blocked-pair diagnostics, paired significance, and per-model lift.
• Split discipline: tune, holdout, and holdback are explicit filters/report labels. The CLI prevents accidental all-split mixing; private prompt_ref storage and when to run hidden splits remain user-owned policy.
• Local grading: deterministic assertions run without model calls.
• Eval hygiene: leakage lint, manifest audit, trigger checks, repeated-run stats, and fixture recommendations.
• Activation: does the skill load on its own? skill-trigger-matrix reports autonomous trigger rates per (agent × model), split by should-fire / should-not-fire.
• Cost as a signal: normalized token/dollar telemetry per run, a suite cost ledger, and lift-per-dollar (cost-summary, token-overhead).
• Interop: Anthropic-style exports, static/served HTML review pages, and Jetty runbook-mode import/export.
• Judge plumbing: judge/rubric assertions can be exported or run through native Claude/Codex/Gemini/Vibe backends (--judge-backend) or a user-supplied --judge-cmd; the harness does not choose a model for you.

Contents

• Questions this helps answer
• Quick start
• Installation
• Manifest format
• Assertions
• Run output contract
• Ablations
• Commands (full detail in docs/commands.md)
• Jetty adapter
• Contributing

Quick start

Requires Python 3.10+ and uv. Install from PyPI:

uv tool install skill-eval-harness

For exact reproducibility, pin the current release: uv tool install skill-eval-harness==0.6.0.

Run these from a skill repo that has evals/shared-benchmark.json:

# 1. Check manifest shape and fixture paths.
skill-benchmark validate evals/shared-benchmark.json

# 2. Emit answer-key-safe task rows for a runner.
skill-benchmark prepare evals/shared-benchmark.json \
  --split tune \
  --runs-per-variant 3 \
  --out /tmp/tasks.jsonl

# 3. Run each task with your agent runner and save:
# eval-runs/latest/<case_id>/<variant>/run-<n>/output.md
# eval-runs/latest/<case_id>/<variant>/run-<n>/metadata.json

# 4. Grade saved outputs. Add --allow-scripts only if you trust repo-owned oracles.
skill-benchmark benchmark evals/shared-benchmark.json \
  --runs eval-runs/latest \
  --split tune \
  --allow-scripts \
  --out benchmark.json

# 5. Open a static review page.
skill-benchmark render-viewer \
  --benchmark benchmark.json \
  --runs eval-runs/latest \
  --out review.html

Expected landmarks:

validate  -> OK: <skill-name> — <case-count> cases, <ablation-count> ablations
prepare   -> /tmp/tasks.jsonl, one JSON object per case/variant/run
benchmark -> benchmark.json with summary, results, and case_flags
viewer    -> review.html with assertion evidence and output previews

benchmark.json records one row per case/variant/run, plus aggregate pass rates, timing/token summaries, and flags for saturated, no-lift, flaky, or with-skill-failed cases. It also carries a reliability block — unbiased pass@k and pass^k per (case, variant) from the repeated runs — beside the paired lift's sign-flip significance, and a trajectory_diff block: per case, over validated experimental pairs, the commands exclusive to one arm across the case's complete repetition set, event-count deltas (steps/commands/tool calls/file reads/file writes), and per-arm skill-load rates — how the arms behaved, beside whether they passed. An arm without non-empty, readable trace evidence blocks its pair with a named reason instead of reading as an empty diff.

Installation

From PyPI

uv tool install skill-eval-harness
skill-benchmark --help
skill-pi-trigger-eval --help

# Pin a release exactly:
uv tool install skill-eval-harness==0.6.0

# One-shot without installing globally:
uvx --from skill-eval-harness skill-benchmark --help

From GitHub

Use this for development snapshots before the next PyPI release:

uv tool install git+https://github.com/adewale/skill-eval-harness.git@main
uvx --from git+https://github.com/adewale/skill-eval-harness.git@main skill-benchmark --help

Upgrading a saved run tree requires more than changing the package pin. Follow the relevant release boundary in docs/upgrading.md before regenerating reports; manifest migration and telemetry migration are separate commands.

The installed commands are:

Command	What it does
skill-benchmark	Validate manifests, prepare tasks, grade outputs, compare variants, run judges, and import/export runner formats.
skill-pi-trigger-eval	Runs Pi without forced --skill and checks whether the model loads the skill from stream events.
skill-trigger-matrix	Measures autonomous skill activation per (agent, model) cell — Claude, Codex, Pi, Vibe, and an offline stub are built in; additional agents add an adapter implementation and one unified backend-registry row.

Local development

git clone https://github.com/adewale/skill-eval-harness.git
cd skill-eval-harness
uv tool install --editable .
skill-benchmark --help

Documentation map

docs/README.md groups these by kind (user journeys, concepts, reference, specs, audits) and holds the convention for adding a new user-journey walkthrough.

File	Use it for
README.md	Manifest shape, run layout, and the command index.
docs/README.md	The docs index: journeys/concepts/reference/specs grouping and the convention for adding a user-journey walkthrough.
docs/commands.md	Full per-command reference: flags, examples, and output shapes for every subcommand.
CHANGELOG.md	Release history and unreleased repo-surface changes.
CONTRIBUTING.md	Local setup, validation commands, and eval-safety rules.
LESSONS_LEARNED.md	Design lessons from the multi-skill saturation work and the roadmap/cost build-out.
docs/architecture.md	How the pipeline fits together: the stages, the runner boundary, the model/variant/run fan-out, and the invariants that keep grading honest.
docs/abstractions.md	What each core object is: manifest, prepared task, run-output contract, assertion result, ResultSet.
docs/typed-python.md	Which Python surfaces ty checks, the boundary inventory, and the drift rules for new modules.
docs/authoring-evals.md	Opinionated workflow/quickstart for writing a new eval suite, including severity and graded assertions.
docs/tuning-skill-activation.md	The activation-tuning loop: trigger cases in both polarities, the (agent, model) trigger-rate matrix, how to read under/over-trigger, and the adapter seam for adding agents.
docs/is-my-skill-worth-its-tokens.md	Keep/trim/cut walkthrough: static footprint (profile-skill) vs. runtime lift-per-token and lift-per-dollar (token-overhead, cost-summary).
docs/gating-ci-on-evals.md	The CI recipe: `report --format junit
docs/did-my-skill-edit-regress.md	The edit → re-run → diff loop: the within-run ablation_regressions block (assertion-level, significance-gated) and cross-iteration render-viewer --previous-workspace diffs over the iteration-N/ convention.
docs/which-model-should-my-skill-target.md	Ranking model tiers by lift: prepare --models fan-out, the by_model / model_analysis blocks, and reading real lift vs. base-model saturation per tier.
docs/why-did-this-run-fail.md	Debugging one failing run: the error-analysis taxonomy + review queue, then the run dir (output.md/metadata.json), mapped to a failure class and a manifest-or-skill decision.
docs/can-i-trust-my-judge.md	Calibrating a judge before believing its numbers: judge-robustness (order-flip + negative controls), judge-alignment (human labels, Cohen's kappa, precision/recall), and compare-judges (does the lift survive a judge swap?).
docs/eval-framework-roadmap-spec.md	The implemented eval-framework roadmap: goals, abstractions, and tests per feature (CF.1–CF.4, buckets 1–4, migration).
docs/migrating-evals.md	Upgrading a manifest between versions (v1 → v2): what migrate stamps and the judgment calls it leaves.
docs/upgrading.md	Version-by-version harness upgrades: saved-run backup, artifact migration, strict input repairs, expected report changes, and rollback.
docs/porting-existing-evals.md	Arriving from another framework: dataset_files + a template case carry the rows across, then the paired baseline, splits, leakage lint, and the audit-manifest punch list supply what the old suite had no slot for.
docs/vocabulary.md	Glossary of harness terms: variants, splits, models, ablations, assertions, severity/oracle tiers, graded scoring, cost telemetry, trace artifacts, agent/judge backends, judge calibration, reliability, contamination, and report flags.
docs/evals-are-not-tests.md	Why a skill eval is not a unit test, and what that changes about reading results.
docs/academic-grounding.md	The research constructs behind the harness's terms, with citations; meshes the workflow, measurement, and theory layers.
docs/jetty-support-spec.md	Jetty payload/import contract and live-token unknowns.
docs/trace-aware-eval-spec.md	Trace artifact contract, shipped v0.4.1 runner support, process/efficiency assertions, and remaining trace work.
docs/telemetry-availability-and-comparability-spec.md	Implemented schema-v3 contract for measured-zero, unavailable, partial, and blocked telemetry/comparisons, including legacy migration.
docs/agent-backend-interface-spec.md	Draft spec for turning Claude/Codex/Gemini/Vibe support into a shared agent backend interface: parity matrix, judge backends, trigger adapters, telemetry, and tool replay.
docs/agent-cli-control-plane.md	The shared native-CLI control plane: process invocation, config isolation, tool policy, final-answer channels, schemas, telemetry, where Claude/Codex/Gemini/Vibe intentionally differ, and the cheap comprehensive live-smoke command.
docs/agent-cli-tradeoffs.md	Claude/Codex/Gemini/Vibe trade-offs: which CLI surfaces are strong or weak and what missing schema/telemetry/prompt controls mean for eval reports.
docs/agent-parity.md	The per-agent support matrix: which answer/judge/trigger surfaces Claude, Codex, Gemini, Vibe, Pi, Jetty, subagent, and the stub each cover, with live-smoke status per backend.
docs/skill-ablation-spec.md	Design spec for materialized (real, altered skill file) ablations: the three-layer model, manifest schema, removal mechanisms, gates, and phased plan.
docs/ablation-study-walkthrough.md + examples/skill-pins.json	A worked ablation study across ten real skills, pinned to exact commit SHAs (+ canonical tree hashes) so it reproduces against the evaluated versions without vendoring any skill content. Includes the replication lesson (2 of 3 single-shot findings refuted at n=5).
docs/repo-effectiveness-audit.md	good-repo audit, score, package metadata fixes, and manual GitHub settings checklist.
docs/correctness-by-construction-audit.md	The closed trigger, experimental-pair, answer-outcome, judge-verdict, prepared-task, Jetty, trace, human-text comparison, and ablation-provenance constructions, their proof tests, and residual risks.
TODO.md	Status tracker: the eval-framework roadmap, remaining Jetty work, Gemini's explicitly gated autonomous-trigger follow-up, the swap:<id> ablation follow-on, and migration/user-journey documentation.
examples/demo-skill/	Self-contained, offline end-to-end example: a tiny synthetic skill, two answer-path materialized ablations, one discovery ablation for trigger examples, and a deterministic stub runner (no model/API). prepare → run-codex → benchmark confirms a regression per answer-path ablation; exercised by tests/test_example_demo.py. Also carries should-fire/should-not-fire trigger cases for skill-trigger-matrix (offline via --agent stub; live smoke via RUN_TRIGGER_SMOKE=1). Start here.
examples/adewale-workspace/	Adewale-specific Pi smoke runner and cross-repo aggregate report (the trigger runners are the top-level skill-pi-trigger-eval and skill-trigger-matrix).
scripts/smoke_supported_clis.py	Opt-in, low-cost smoke across native Claude/Codex/Gemini/Vibe answer paths and Pi trigger path using a disposable demo-skill eval.
tests/test_skill_benchmark.py	Executable examples for grading, leakage lint, script assertions, judge commands, Jetty export/import, trace artifacts, and trigger detection.

Manifest format

Each skill repo owns an evals/shared-benchmark.json manifest. Add a harness block so readers know which external harness/version to install.

{
  "version": 1,
  "skill_name": "good-pr",
  "harness": {
    "name": "skill-eval-harness",
    "url": "https://github.com/adewale/skill-eval-harness",
    "version": ">=0.6.0"
  },
  "skill_paths": ["skills/good-pr/SKILL.md"],
  "variants": ["with_skill", "without_skill"],
  "optional_variants": ["old_skill"],
  "split_policy": {
    "tune": "Visible cases used during iteration.",
    "holdout": "Hidden cases scored only at end-of-round or merge.",
    "holdback": "Examples not exposed in skill/docs/eval descriptions until after scoring."
  },
  "cases": [
    {
      "id": "pos-security-meaningless-test",
      "split": "tune",
      "kind": "pr-review",
      "domain": "pull-request-quality",
      "difficulty": "core",
      "trigger_type": "explicit",
      "success_goals": ["outcome", "style"],
      "prompt": "Security fix PR includes `expect(result).toBeDefined()` as the only auth-bypass test...",
      "files": ["fixtures/security-pr/diff.patch"],
      "expected_behavior": ["Flag the weak test and require regression proof."],
      "assertions": [
        {"name": "detect-weak-test", "type": "contains_any", "values": ["weak", "toBeDefined"]},
        {"name": "qualitative-review", "type": "judge", "rubric": ["Specific", "maintainer-friendly"]}
      ],
      "tags": ["security", "testing"]
    }
  ],
  "ablations": [
    {
      "id": "no-regression-proof",
      "removed_component": "regression-proof requirement",
      "expected_regressions": ["Accepts weak tests that still pass without the fix"]
    }
  ]
}

Splits

Split	Purpose	Prompt storage
tune	Visible cases used while editing the skill and evals.	Inline prompt is fine.
holdout	Hidden cases scored at end-of-round or merge.	Prefer private prompt_ref.
holdback	Not shown in skill/docs/evals until after scoring; detects memorization.	Prefer private prompt_ref and ignored answer keys.

prepare fails on missing hidden prompts unless --allow-missing-prompts is used for dry-run planning.

Use optional files for fixture-backed evals. Paths are relative to the manifest's evals/ directory, validated by validate, and emitted by prepare as absolute input_files for the runner.

Further optional manifest surfaces (each with a behavior-preserving default; see docs/migrating-evals.md):

• version: 1 or 2 — skill-benchmark migrate upgrades 1 → 2 by stamping the defaults explicitly.
• judge: {"model": "..."} — the default judge model for the judge command; audit-manifest flags judge-is-model-under-test (fatal under --strict-judge).
• datasets + a case template: fan one case template over rows with {key} placeholder filling and stable ids (<case>-<row id|index>); leakage lint runs per materialized case.
• turns on a case: a scripted multi-turn sequence; each turn's assertions grade that turn's transcript entry (turn-<n>/output.md), case-level assertions grade the final answer.
• YAML manifests: a .yaml manifest (plus dataset_files mapping dataset ids to JSONL row files) compiles to the same shape in memory — validation, lint, and grading are identical.
• Reference floors: reference_score (0-1) / reference_graded_score (1-5).

Assertions

Objective assertion types:

Type	Checks
contains	One substring is present.
contains_any	At least one substring is present.
contains_all	Every listed substring is present.
excludes_any	No listed substring is present.
regex	Regex matches output.
not_regex	Regex does not match output.
file_exists	A file exists relative to the run directory.
json_field_equals	A JSON field equals an expected value.
golden_output	Output (or a named artifact) equals a reference file; explicit normalization (exact default, trim, text); unified diff as failure evidence.
similarity	difflib ratio against an expected string with a threshold (default 0.8), emitting a score. mode: "embedding" uses cosine similarity behind the opt-in --embed-cmd.
structured_output	JSON (an artifact via path, or extracted from the output) validates against a deterministic JSON-Schema subset (type/properties/required/items/enum/const/minItems/maxItems).
script	Opt-in deterministic oracle command against the output directory. A stdout line like {"score": 6, "max_score": 7} feeds the graded channel; exit code still decides pass/fail.
skill_invoked	Trace/process check that the runner loaded the skill, or did not, as expected.
command_ran / command_not_ran	Trace/process checks over normalized command events.
command_order	Trace/process check that commands appeared in a required order.
tool_call	A tool call matching tool/pattern occurred (with min_count/max_count bounds), or an ordered order list of calls. BFCL-style set relations over completed-call tool names (exact, case-insensitive — not substring): expected_no_call (the named tool, or any name matching pattern, must never have been observed—even as started, failed, or in progress), required_calls (an order-independent subset of completed tool names that must all appear, extras allowed), call_set (an exact multiset of completed tool names—same names and multiplicities, no unexpected named calls). Use pattern/order/command_ran for regex or command-text matching. Positive selectors match completed call inputs, never outputs.
tool_count_le / no_repeated_command_loop	Trace/process budgets for tool use and thrashing.
total_tokens_le / elapsed_seconds_le / command_count_le	Efficiency checks over metrics.json, metadata.json, or normalized events.

Assertion objects are closed contracts, including nested graded_dimensions and dynamic_rubric objects: unknown fields and fields that do not apply to the selected assertion type are validation errors, so a misspelled severity or path cannot silently change the grader. A golden_output reference must already be a regular file. A local script oracle must live in a dedicated subdirectory (for example oracles/check.py); the harness binds that oracle tree into the eval-contract digest and rejects symlinks. Changing an imported helper or data file therefore invalidates stale prepared runs, while generated files beside the manifest cannot make the contract self-referential.

Human-readable answer assertions (contains, contains_any, contains_all, excludes_any, regex, not_regex, and similarity) compare through the versioned rendered-v1 view by default. The raw output.md is never rewritten: the comparison view applies NFC canonical normalization and removes only a narrow allow-list of zero-width, non-ordering controls: U+200B ZERO WIDTH SPACE, U+2060 WORD JOINER, and U+FEFF ZERO WIDTH NO-BREAK SPACE. Controls that can change visible glyph or