Agent Reliability — MCP server

> Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence

A remote MCP server over a curated knowledge graph. Every claim it

returns is bound to a registered source: the tools hand back claims *with*

their citations and a confidence value, so an agent can show its work

instead of asserting.

Nothing to install. It is a hosted streamable-HTTP endpoint:

https://agentreliability.dev/mcp

Also listed on Smithery.

Add it to a client

Claude Code

claude mcp add --transport http agent-reliability https://agentreliability.dev/mcp

Claude Desktop / any client reading `mcpServers`

{
  "mcpServers": {
    "agent-reliability": {
      "type": "streamable-http",
      "url": "https://agentreliability.dev/mcp"
    }
  }
}

No API key, no account, no auth. Read-only.

Check it answers, without any client at all:

curl -s https://agentreliability.dev/mcp \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -H 'mcp-protocol-version: 2025-06-18' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

Tools

Eight, each with an `outputSchema`, each returning `structuredContent`.

The intended path is `get_overview` → `search` or `answer` → `get_entity`

→ `get_related`. `get_overview` exists because an agent that has just

arrived needs to know whether this corpus can help *before* it spends a

call guessing.

What is in the corpus

Subject matter: evals and benchmarks (GAIA, AgentBench, Inspect), LLM-as-judge and its failure modes, Goodhart and benchmark contamination, fault injection and chaos testing, approval gates and autonomy levels, grounding and faithfulness.

Questions it is built to answer

• *How do I tell a real eval from a benchmark my agent has memorised?*
• *What does calibration mean for an LLM judge, and how is it measured?*
• *Which failure modes does fault injection actually catch?*

What an answer actually looks like

A real call against the live endpoint — `answer` with

*"how do I tell a real eval from benchmark contamination"* — returns this `structuredContent`, trimmed:

{
  "answered": true,
  "entity": {
    "id": "agent-reliability-glossary",
    "name": "Agent reliability glossary",
    "evidence_tier": "secondary",
    "confidence": 0.85,
    "last_verified": "2026-08-08",
    "canonical_url": "https://agentreliability.dev/k/agent-reliability-glossary"
  },
  "claims": [
    {
      "text": "An eval is a structured, repeatable test that measures an LLM or LLM-based system against a defined dimension; frameworks package evals as registries of reusable templates.",
      "sources": [{ "title": "openai/evals — framework for evaluating LLMs and LLM systems" }]
    }
  ]
}

Note what travels with the answer: the evidence tier, a confidence,

the date it was last verified, and the source behind the claim — not

as prose an agent has to parse, but as fields it can act on. An agent can

decline to use a weak claim, or cite the primary source directly.

When the corpus cannot answer, `answered` is `false`. It does not

improvise, and the miss is recorded so the gap can be filled.

Machine-readable surfaces

The MCP endpoint is one of several. The same corpus is served as plain

files an agent can read directly:

Each knowledge object has a human page and a machine twin at the same id,

with a canonical URL that agrees across all of them.

Behaviour worth knowing before you integrate

• `POST` only. Every other method answers `405` with an `Allow: POST, OPTIONS` header.
• Rate limit: 120 requests per minute per client, counted in a shared

store, published on every response as `RateLimit-Limit`,

`RateLimit-Remaining` and `RateLimit-Reset` (all three exposed via CORS).

It fails open: if the store is unreachable the request is served.

• Malformed input gets a spec-correct JSON-RPC error — `-32700` for

unparseable bodies, `-32602` for an unknown tool — never an HTML error page.

• Request bodies are capped and validated before transport.

Privacy

No accounts, no cookies, no ads. Usage is measured in aggregate with

daily-rotating hashed identifiers and a 200-day retention; raw IPs are

never stored. Full policy: PRIVACY.md.

Provenance and licence

Knowledge content is CC-BY-4.0: use it, cite it. The source registry is

public precisely so a claim can be checked rather than trusted —

`get_sources` returns what any given claim rests on.

Claims carry an evidence tier and a `last_verified` date. Where the

evidence is weaker, the object says so rather than rounding up.

How it is built

Compiled and served by Citarium, a source-available framework for turning a

knowledge graph into a website, an API, an MCP server and agent-readable

files from a single source, under external evaluation.

The framework's code is licensed under the Business Source License 1.1 and

its repository is not public. What is public — and what actually matters for

trusting an answer — is this server, the corpus it serves, and the registered

source behind every claim: `get_sources` returns what any given claim rests

on, so it can be checked rather than trusted.

This repository is the server's public face: its manifest and its

documentation. The corpus itself lives at agentreliability.dev.