trackmcp
Back to directory
AlligatorC0der

conkurrence

View on GitHub

AI evaluation toolkit — measure inter-rater agreement across multiple LLM providers

0 stars DockerfileOthers Updated Apr 8, 2026

Documentation

ConKurrence

One command. Find out if your AI agrees with itself.

ConKurrence is a statistically validated consensus measurement toolkit for AI evaluation pipelines. It uses multiple AI models as independent raters, measures inter-rater reliability with Fleiss' kappa and bootstrap confidence intervals, and routes contested items to human experts.

Install

bash
npm install -g conkurrence

MCP Server

Use ConKurrence as an MCP server in Claude Desktop or any MCP-compatible client:

bash
npx conkurrence mcp

Claude Desktop Configuration

Add to your `claude_desktop_config.json`:

json
{
  "mcpServers": {
    "conkurrence": {
      "command": "npx",
      "args": ["-y", "conkurrence", "mcp"]
    }
  }
}

Claude Code Plugin

code
/plugin marketplace add AlligatorC0der/conkurrence

Features

  • Multi-model evaluation — Run your schema against Bedrock, OpenAI, and Gemini models simultaneously
  • Statistical rigor — Fleiss' kappa with bootstrap confidence intervals, Kendall's W for validity
  • Self-consistency mode — No API keys needed; uses the host model via MCP Sampling
  • Schema suggestion — AI-powered schema design from your data
  • Trend tracking — Compare runs over time, detect agreement degradation
  • Cost estimation — Know the cost before running

MCP Tools

ToolDescription
`conkurrence_run`Execute an evaluation across multiple AI raters
`conkurrence_report`Generate a detailed markdown report
`conkurrence_compare`Side-by-side comparison of two runs
`conkurrence_trend`Track agreement over multiple runs
`conkurrence_suggest`AI-powered schema suggestion from your data
`conkurrence_validate_schema`Validate a schema before running
`conkurrence_estimate`Estimate cost and token usage

Links

License

BUSL-1.1 — Business Source License 1.1

Frequently asked questions

What is conkurrence?

conkurrence is AI evaluation toolkit — measure inter-rater agreement across multiple LLM providers

How do I install conkurrence?

Open the GitHub repository and follow its README. Most MCP servers are added to your client's MCP config, then called by your agent.

Is conkurrence open source?

Yes — it is hosted on GitHub at https://github.com/AlligatorC0der/conkurrence.

Related MCP tools

modelcontextprotocolservers

Model Context Protocol Servers

90,051 TypeScript
colbymchenrycodegraph

Pre-indexed code knowledge graph, auto syncs on code changes, for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, CoPilot, and Hermes Agent — fewer tokens, fewer tool calls, 100% local

69,582 C
aaif-goosegoose

an open source, extensible AI agent that goes beyond code suggestions - install, execute, edit, and test with any LLM

53,903 Rust
mcpacpai+1
metabasemetabase

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

49,091 Clojure
analyticsbusinessintelligencedashboard+15
bytedanceUI-TARS-desktop

The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra

38,840 TypeScript
agentvlmvision+11
topoteretescognee

Cognee is the open-source AI memory platform for agents. Give your AI agents persistent long-term memory across sessions with a self-hosted knowledge graph engine.

30,433 Python
agent-memoryagent-skillsai+16

Run your own MCP server? See who uses it and what to fix.

Measure it with TrackMCP