The eval.yaml schema¶

A single eval.yaml in your project root drives everything. It is parsed into an EvalConfig and validated at load time. Every top-level key is optional and has a sensible default — a minimal config is just a name, what to execute, a dataset, and one judge.

Describe what, not where

eval.yaml describes what to evaluate. The execution backend (Local, Harbor, EvalHub) is always a CLI flag (--runner), never a config key — so the same file runs unchanged everywhere.

Every top-level key is optional. name and description are the only non-deprecated top-level scalars. LLM marks natural-language fields interpreted by agents & judges, PY marks Python code or expressions, and [] marks a list.

Define · what to evaluate

dataset

Where cases live and what they contain

path schemaLLM workspace.files[]

generation

How /eval-dataset sources cases

strategy contextLLM seeds[]

inputs.tools[]

Tool interception for headless runs

matchLLM promptLLM prompt_fileLLM

outputs[]

Artifacts and tool calls to collect

path tool schemaLLM batch_pattern types

Execute · how it runs

execution

What runs (skill or prompt), per case or batch

mode skill promptLLM arguments timeout max_budget_usd parallelism env

runner

Agent runtime and knobs

type effort settings plugin_dirs[] env system_promptLLM command workspace_mode

models

Model per role (CLI flags override)

skill subagent judge hook

permissions

Tool allow / deny for headless runs

allow[] deny[]

hooks

Lifecycle shell commands

before_all[] before_each[] after_each[] before_scoring[] after_all[] before_report[]

skilldeprecated

Top-level skill: — use execution.skill

Score · how it's judged

judges[]

How each case is scored — four judge types

name descriptionLLM ifPY checkPY promptLLM prompt_fileLLM llm_rubricLLM builtin modulePY functionPY context[] model arguments samples score_range feedback_type

thresholds

Regression gates per judge

min_mean min_pass_rate min_win_rate

reward

Collapse judges into an RL scalar in [0, 1]

judge normalize formulaPY weights gate score_range raw[]

Observe · what's captured

mlflow

Experiment tracking — presence opts in

experiment tracking_uri tags

traces

Execution data captured for judges

stdout stderr events metrics

Top-level keys¶

Key	Purpose	Reference
`name`	Experiment / run name (defaults to the file stem)	(inline)
`description`	Human-readable description	(inline)
`execution`	What to run and how cases are processed	execution
`runner`	Agent runtime + runtime-specific knobs	runner
`models`	Model per role: skill, subagent, judge, hook	models
`permissions`	Tool allow/deny for headless runs	permissions
`mlflow`	Experiment tracking (opt-in)	mlflow
`dataset`	Where cases live and what they contain	dataset
`generation`	How `/eval-dataset` sources cases	generation
`inputs`	Tool interception handlers (`inputs.tools`)	inputs.tools
`outputs`	Artifacts / tool calls to collect	outputs
`traces`	Which execution data to capture	traces
`hooks`	Lifecycle shell hooks	hooks
`judges`	How each case is scored	judges
`thresholds`	Regression gates per judge	thresholds
`reward`	Collapse judges into an RL reward scalar	reward
`skill`	Deprecated — use `execution.skill`	(see below)

skill: at the top level is deprecated

A top-level skill: still works but is auto-normalized into execution.skill with a deprecation warning. Always author new configs with execution.skill.

Two minimal configs¶

Which keys you set depends on whether you're testing a skill or a capability. See the execution model for the difference.

Skill modePrompt mode

name: my-skill-eval

execution:
  mode: case
  skill: my-skill
  arguments: "{prompt}"

dataset:
  path: eval/dataset/cases
  schema: "Each case has an input.yaml with a 'prompt' field."

judges:
  - name: output_quality
    prompt: "Score the output 1-5 for completeness and accuracy."

name: docs-navigation-eval

execution:
  mode: case
  prompt: "{{ input.prompt }}"

runner:
  workspace_mode: repo   # navigate the real repository

dataset:
  path: eval/dataset/cases
  schema: "Each case has an input.yaml with a 'prompt' field."

judges:
  - name: used_docs
    builtin: consulted_docs

A fully annotated config¶

The repository's root eval.yaml is the canonical, heavily-commented reference — every block with inline comments and commented-out variants for all four judge types, tool interception, batch_pattern, and thresholds. It's the best single file to copy from.

eval.yaml (excerpt)

name: my-skill-eval
description: Evaluate the main skill pipeline

execution:
  mode: case              # per-case (default) or batch
  skill: my-skill-name    # skill to test (use `prompt:` for prompt mode)
  arguments: "{prompt}"   # resolved per case from input.yaml fields

runner:
  type: claude-code       # claude-code | cli | responses-api
  # effort: high          # low | medium | high | xhigh | max

models:
  skill: claude-opus-4-6  # required (or pass --model)
  judge: claude-opus-4-6  # used by LLM and pairwise judges

permissions:
  deny:
    - "mcp__*"            # block all MCP tools during eval

mlflow:
  experiment: my-skill-eval   # opt-in: omit the block to disable tracking

dataset:
  path: eval/dataset/cases
  schema: |
    Each case has input.yaml (a 'prompt' field) and reference.md (gold output).

outputs:
  - path: artifacts
    schema: "One markdown file per case, named NNN-slug.md."

traces:
  stdout: true
  stderr: true
  events: false
  metrics: true

judges:
  - name: has_content
    check: |
      content = outputs["main_content"]
      if len(content.strip()) < 100:
          return False, f"Output too short ({len(content.strip())} chars)"
      return True, f"Output has {len(content.strip())} chars"

  - name: output_quality
    prompt: "Score 1-5 vs the reference for completeness, clarity, accuracy."

thresholds:
  has_content: { min_pass_rate: 1.0 }
  output_quality: { min_mean: 3.5 }

Conventions¶

Schema fields are natural language. dataset.schema and outputs[].schema are documentation for the LLM agents and judges — scripts operate on file paths, not a parsed spec. There are no hardcoded field names.
Load-time validation is strict. Mutually-exclusive keys (skill + prompt), invalid enums (execution.mode), and malformed reward formulas fail at load, not mid-run.

Per-key reference¶

execution — mode, skill/prompt, arguments, timeout, budget, parallelism, env
runner — type, effort, settings, plugin_dirs, env, system_prompt, command, workspace_mode
models — skill, subagent, judge, hook roles and precedence
permissions — allow/deny patterns and the path-based compiler
mlflow — experiment, tracking_uri, tags
dataset — path, schema, workspace.files
generation — strategy, context, seeds
inputs.tools — tool interception handlers
outputs — path vs tool, schema, batch_pattern, types
traces — stdout, stderr, events, metrics
hooks — before/after all/each, before_scoring
judges — the four judge types and all fields
thresholds — min_mean, min_pass_rate, min_win_rate
reward — single-judge and formula reward modes