Get Started¶

The harness ships as a Claude Code plugin. Once it's installed, you drive it entirely through slash commands (/eval-*). This section takes you from zero to a scored HTML report.

The shortest path¶

Analyze — point the harness at a skill and let it write eval.yaml. /eval-analyze --skill my-skill
Build a dataset — generate test cases that match the config. /eval-dataset
Run — execute, score with judges, and build the HTML report. /eval-run --model opus

That's it — /eval-setup and /eval-mlflow are optional (dependencies auto-install, and MLflow logging is opt-in).

How the pieces fit¶

graph TD
    subgraph required ["Required for a first run"]
        A["/eval-analyze<br/>writes eval.yaml"] --> D["/eval-dataset<br/>writes test cases"]
        D --> R["/eval-run<br/>execute + score + report"]
    end
    subgraph optional ["Optional"]
        S["/eval-setup<br/>env + MLflow"] -.-> A
        R -.-> V["/eval-review<br/>human feedback"]
        R -.-> O["/eval-optimize<br/>auto-refine"]
        R -.-> M["/eval-mlflow<br/>log + trace"]
    end

In this section¶

Installation & setup

Add the plugin, install dependencies, and configure API keys and MLflow.
Your first eval (skill mode)

Analyze a skill, generate a dataset, run it, and read the report.
Your first agentic-docs eval

Test whether an agent can navigate and correctly use your documentation.
Reading the report

Understand scores, per-case detail, diffs, and cost.

New to the terminology?

Two words are worth pinning down before you start: a runner is the agent runtime (claude-code, cli, responses-api), while an execution backend is where it runs (Local, Harbor, EvalHub). See the Glossary.