Skip to content

Docs UX preview for opendatahub-io/agent-eval-harness — hosted on a personal fork to show owners a clearer landing experience. Not a long-lived product fork; install and clone URLs still point upstream.

Make agent performance measurable — and improvable

Evaluate skills and agent capabilities with one declarative eval.yaml: analyze, generate cases, run, judge, trace in MLflow, then optimize. Same config on your laptop, Harbor containers, or EvalHub.

Get started eval.yaml reference

Python 3.11+ Apache-2.0 Claude plugin MLflow Harbor

Made at Red Hat

How the loop works

Five stages. Only analyze → dataset → run are required for a first score.

  • 1 · Analyze

    Point /eval-analyze at a skill or a prompt brief. The harness writes eval.yaml with judges, schema, and thresholds.

  • 2 · Dataset

    /eval-dataset fills cases from your schema — or bring your own cases/ tree with gold references.

  • 3 · Run & judge

    /eval-run executes on Claude Code (or another runner), scores with LLM + code judges, and emits a rich HTML report.

  • 4 · Trace

    Optional /eval-mlflow syncs metrics, artifacts, and hierarchical GenAI traces for every case.

  • 5 · Optimize

    /eval-optimize proposes skill fixes from failures and re-runs so you keep only real gains.

See the full pipeline guide

What you get

  • Skill or prompt mode

    Test a packaged skill (execution.skill) or agent capability directly (execution.prompt) — including agentic documentation checks.

    Execution model

  • LLM + code judges

    Built-in judges, inline Python checks, rubrics, pairwise A/B, and N-sample stability — all in one config.

    Judges & scoring

  • One config, three backends

    Local subprocess, Harbor (Podman / OpenShift), or EvalHub — backend is a CLI flag, never baked into eval.yaml.

    Execution backends

  • Any agent runtime

    Claude Code out of the box; bring OpenCode or a custom CLI / Responses API runner when you need it.

    Runners

  • Reward API for RL

    Collapse judges into a [0, 1] reward for GRPO-style training via Harbor / NeMo Gym / SkyRL.

    Reward API

  • MLflow-native

    Experiments, datasets, hierarchical traces, and feedback sync — opt in with one mlflow: block.

    Tracing

Choose your path

Path Use it when Start
Claude Code plugin You want slash commands in an existing project claude plugin install agent-eval-harness@opendatahub-skills
Local clone You are hacking on the harness itself git clone https://github.com/opendatahub-io/agent-eval-harness
Harbor / OpenShift You need containerized, reproducible trials Running on Harbor 
# Upstream install (proposal preview does not change this)
claude plugin install agent-eval-harness@opendatahub-skills
/eval-setup
/eval-analyze --skill my-skill
/eval-dataset
/eval-run --model opus

Explore the docs

  • Get Started

    Install and run your first evaluation end to end.

    Get Started

  • Guides

    Task-oriented how-tos for every skill and backend.

    Guides

  • Concepts

    Execution model, judges, rewards, and tracing.

    Concepts

  • Cookbook

    Worked configs for common evaluation scenarios.

    Cookbook

  • Reference

    eval.yaml schema, CLI, config fields, and glossary.

    Reference