pip install agent-regress-cli

p=0.003, Cohen's d=-0.61 -> REGRESSED (deploy blocked)
p=0.410, Cohen's d=0.021 -> STABLE (safe to ship)

pip install agent-regress-cli
# or
uv add agent-regress-cli
# or, from Node/npx (thin wrapper around the same Python CLI)
npx agent-regress-cli

pip install agent-regress-cli

agent-regress compare \
  --version-a-results v1_scores.json \
  --version-b-results v2_scores.json \
  --metric tool_accuracy

# ============================================================
# agent-regress Report -- tool_accuracy
# ============================================================
# Verdict:    REGRESSED
# p-value:    0.0000
# Cohen's d:  -2.193
# 95% CI:     [-0.213, -0.148]
#
# Version A:  0.8470 +/- 0.0525  (n=50)
# Version B:  0.6685 +/- 0.1025  (n=50)
# Delta:      -0.1786
# ============================================================

from agent_regress import compare

# Any callable that takes a test case dict and returns a score 0.0-1.0
def agent_v1(test_case: dict) -> float:
    ...  # your existing agent

def agent_v2(test_case: dict) -> float:
    ...  # your updated agent

test_suite = [
    {"query": "find SKU for order 8823", "expected": "SKU-4492"},
    # ... more test cases
]

report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    metric="tool_accuracy",  # use any name except "accuracy" when agents return floats
)

print(report)           # structured output with p-value, CI, effect size
report.assert_stable()  # raises AssertionError if behavior regressed

from agent_regress import compare, exact_match_scorer, f1_scorer

# exact_match_scorer: 1.0 if str(output).strip() == str(expected).strip()
# f1_scorer: token-level F1 (multiset — handles repeated tokens correctly)
report = compare(
    version_a=agent_v1,
    version_b=agent_v2,
    test_suite=test_suite,
    n_runs=50,
    scorer=exact_match_scorer,  # test_case must have an "expected" key
)

def my_scorer(output: str, test_case: dict) -> float:
    return 1.0 if output.strip() == test_case["expected"] else 0.0

report = compare(..., scorer=my_scorer)

pip install "agent-regress-cli[mcp]"

{
  "mcpServers": {
    "agent-eval": {
      "command": "uvx",
      "args": ["--from", "agent-regress-cli", "agent-regress-mcp"]
    }
  }
}

run(["compare", "--version-a-results", "a.json", "--version-b-results", "b.json", "--metric", "accuracy"])

============================================================
agent-regress Report -- tool_accuracy
============================================================
Verdict:    REGRESSED
p-value:    0.0031
Cohen's d:  -0.610
95% CI:     [-0.221, -0.067]

Version A:  0.8400 +/- 0.0601  (n=50)
Version B:  0.7000 +/- 0.0903  (n=50)
Delta:      -0.1400
============================================================

AssertionError: REGRESSED: tool_accuracy dropped 16.7%
(p=0.003, Cohen's d=-0.61, 95% CI [-0.22, -0.07])
Version A: 0.840 +/- 0.060  (n=50)
Version B: 0.700 +/- 0.090  (n=50)

Verdict:    STABLE
p-value:    0.4100
Cohen's d:  0.021

# test_regression.py -- add to your existing test suite
from agent_regress import compare

def test_no_regression():
    report = compare(
        version_a=production_agent,
        version_b=staging_agent,
        test_suite=load_test_suite(),
        n_runs=50,
    )
    report.assert_stable(
        p_threshold=0.05,  # act on changes at p = 0.2.

See [docs/statistical-methods.md](docs/statistical-methods.md) for the full methodology.

---

## Benchmarks

Statistical test overhead is the time to run the comparison itself, not the agent calls. Agent calls are the bottleneck; the statistics are not.

Measured on Apple M3 Pro, Python 3.14, scipy 1.15, numpy 2.2:

| Operation | n=50 per version | n=1,000 per version |
|---|---|---|
| Mann-Whitney U | **0.34ms** | **0.47ms** |
| Bootstrap CI (1,000 resamples) | **26ms** | **31ms** |
| Full compare() statistical overhead | **~27ms** | **~32ms** |

See [docs/benchmarks.md](docs/benchmarks.md) to reproduce.

---

## Integration matrix

| Framework | Status | Install |
|---|---|---|
| LangGraph | Shipped (v0.1) | `pip install agent-regress-cli[langgraph]` |
| OpenAI Agents SDK | Shipped (v0.1) | `pip install agent-regress-cli[openai-agents]` |
| CrewAI | Shipped (v0.1) | `pip install agent-regress-cli[crewai]` |
| LangChain LCEL | Shipped (v0.1) | `pip install agent-regress-cli[langchain]` |
| AutoGen | Planned (v0.3) | |
| Vercel AI SDK (TypeScript) | Planned (v0.4) | |

Comparing two *installed versions* of the same framework (rather than two
in-process configurations)? See
[docs/cross-version-comparison.md](docs/cross-version-comparison.md) for the
`subprocess_runner()` pattern.

> [!WARNING]
> The `[crewai]` extra: CrewAI's own memory/knowledge/RAG backend can pull in ChromaDB, which currently has an unpatched critical CVE ([GHSA-f4j7-r4q5-qw2c](https://github.com/advisories/GHSA-f4j7-r4q5-qw2c)) affecting any ChromaDB server run with `trust_remote_code=True` and exposed to the network. `agent-eval` never starts, configures, or exposes a ChromaDB server itself, so this only matters if your own `Crew` does — don't run a network-exposed ChromaDB instance with `trust_remote_code=True` until a fix ships.

---

## Standard benchmarks

Agent Evaluation ships harnesses for the three standard agent benchmarks:

**Tau-bench pass^k** measures reliability across k independent attempts. Single-run benchmarks miss degradation: an agent that succeeds 60% of the time at k=1 reaches 99.93% at k=8. The k=1 vs k=8 curve is the signal.

**GAIA Level 1-3 split** stratifies by task difficulty. Overall accuracy hides per-difficulty regressions: a prompt change that helps Level 1 often hurts Level 3.

**SWE-bench scaffold score** isolates framework contribution from model contribution.

See [leaderboard/README.md](leaderboard/README.md) to submit results.

---

## Try it in Docker

Starts two services:

- **web** (`http://localhost:8080`) — leaderboard UI served by `web/serve.py`, reading `leaderboard/results/*.json`
- **example** — runs `examples/01-basic-comparison/example.py` and prints the comparison report to stdout

Good for verifying the install works and seeing the leaderboard UI before wiring agent-regress into your own agent.

---

## Security

- **Supply chain:** Releases are built and published from a GitHub Actions workflow, signed with Sigstore, and ship with a CycloneDX SBOM attached to every GitHub Release. (No SLSA provenance attestation is generated yet — that would require adopting `slsa-framework/slsa-github-generator`.)
- **Vulnerability scanning:** Trivy scans on every CI run (HIGH/CRITICAL only, exit on unfixed). CodeQL static analysis on every push.
- **Dependency pinning:** Dependabot keeps all GitHub Actions and Python dependencies current.
- **Disclosure:** [SECURITY.md](SECURITY.md) — report vulnerabilities privately via GitHub Security Advisories.

---

## Leaderboard

The `leaderboard/` directory version-controls Tau-bench pass^k, GAIA, and SWE-bench results across models and frameworks. Submit by opening a PR with a JSON file matching `leaderboard/schema.json`. Results are independently reproduced before merging.

See [leaderboard/README.md](leaderboard/README.md).

---

## FAQ

**What is agent-eval, and what makes it different from a normal LLM eval framework?**

Agent Evaluation is a statistics library for detecting whether an agent's behavior actually changed between two versions. Run the same test suite 50 times on version A and 50 times on version B, and it reports a p-value (Mann-Whitney U), an effect size (Cohen's d), and a bootstrap 95% confidence interval on the score delta. Most eval frameworks check whether a single response clears a fixed quality threshold. Agent Evaluation instead answers a distributional question: did the score distribution shift significantly, or is a change just LLM-run-to-run noise.

**How do I install it, and which platforms does it support?**

`pip install agent-regress-cli` or `uv add agent-regress-cli`. It requires Python 3.10 through 3.13 (per the classifiers in `pyproject.toml`) and has no OS-specific code, so it runs anywhere those Python versions run. A Node/npx wrapper (`npx agent-regress-cli`, also published as `agent-regress-cli` on npm) is also available -- it shells out to this same Python package, so a Python toolchain (or `uv`/`pipx`, which can run it ephemerally without a manual `pip install` step) still needs to be available; it prints an actionable error if neither is found.

**How does it compare to DeepEval, Promptfoo, or Braintrust?**

The full breakdown is in the [comparison table](#why-not-deepeval-promptfoo-or-braintrust) above. In short: DeepEval, Promptfoo, and Braintrust all test whether an individual response clears a fixed quality bar. None of the three report a p-value, an effect size, or a bootstrap confidence interval on whether behavior shifted between two versions, which is the specific statistical question agent-eval is built to answer.

**I ran a comparison and got a warning about insufficient statistical power, or a verdict of INSUFFICIENT_DATA. What does that mean?**

The library warns (but does not fail) when either version has fewer than 50 runs, since that is the sample size needed for reliable detection of a moderate effect (Cohen's d of 0.2) at 80% power. Below 10 runs per version, it returns `INSUFFICIENT_DATA` instead of a REGRESSED/STABLE/IMPROVED verdict, since the sample is too small to trust any statistical conclusion. Re-run with `n_runs=50` or higher for a verdict you can act on.

**Does agent-eval call my LLM or manage API keys for me?**

No. `compare()` takes two callables you provide, `version_a` and `version_b`, and runs your existing agent code against your test suite. Agent Evaluation never makes a model call itself, and the stats module (`src/agent_regress/stats/`) is required to stay pure Python and scipy with no LLM calls, so the statistical core has no network dependency and nothing to configure credentials for.

**Which agent frameworks does it integrate with today?**

LangGraph, the OpenAI Agents SDK, CrewAI, and LangChain LCEL are shipped as of v0.1 (see the [integration matrix](#integration-matrix) above), each installable as an extra, e.g. `pip install agent-regress-cli[langgraph]`. AutoGen and a Vercel AI SDK (TypeScript) integration are planned but not yet shipped.

**Can I use agent-eval commercially, and what license is it under?**

Yes. It is licensed under Apache License 2.0, which permits commercial use, modification, and distribution, and includes an explicit patent grant. You need to preserve the copyright and license notices and state any changes you make; there is no warranty. See [LICENSE](LICENSE) for the full text.

---

## Contributing

- Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a PR
- Good first issues are labeled in GitHub
- Stats module (`src/agent_regress/stats/`) must stay pure Python + scipy — no LLM calls, ever
- All PRs require 95% coverage on `stats/`, 80% overall

GitHub Discussions for design questions.

Apache 2.0. Contributions welcome.

---

## Cite this work

If you use Agent Evaluation in research, please cite:

---

*Built by Rudrendu Paul and Sourav Nandy*