- Evaluate selection, argument validity, execution, recovery, completion, and safety separately.
- Record the client, model, catalog, schema, server, and test versions.
- Publish sample size and limitations so the benchmark remains credible.
MCP tool-selection evaluation measures whether an agent chooses the right tool for a user intent and uses it successfully. Separate tool choice from argument validity, tool execution, recovery, and workflow completion so a single score does not hide where the system failed.
Build a labeled intent dataset
Start with real or carefully constructed user intents that your server is expected to support. For each intent, define one expected tool or an acceptable set of tools, required arguments, disallowed side effects, and the terminal outcome.
Score the stages separately
- Selection accuracy: did the agent choose an acceptable tool?
- Argument validity: did the call satisfy the schema?
- Execution success: did the tool return an acceptable result?
- Recovery: did the agent respond correctly to an error or missing input?
- Workflow completion: did the end-to-end task reach its expected outcome?
- Safety: did the agent avoid disallowed or destructive tools?
Keep the test conditions fixed
Record client, model, tool catalog version, schema version, prompt, system instructions, temperature or determinism settings, server version, and test date. A change in any of these can change the result. If the test is synthetic, say so clearly.
Use a simple scorecard
- Selection accuracy = acceptable tool selections divided by eligible intents.
- Argument validity = valid tool calls divided by selected tool calls.
- Execution success = successful tool results divided by executed tool calls.
- Workflow completion = completed workflows divided by started workflows.
- Safety rate = safe runs divided by eligible intents.
Keep the component scores visible. A single composite score can be useful for a trend line, but it should never replace the underlying measures. A server can select correctly and still fail because a dependency is slow; it can also complete a task while selecting a dangerous tool along the way.
Run description and schema experiments carefully
When testing a tool-description or schema change, change one meaningful variable at a time, keep the intent set stable, and compare the result with a baseline. Look for higher selection accuracy and completion without a rise in unsafe calls, invalid arguments, or retries.
Publish limitations with every benchmark
- Number and source of intents.
- Clients, models, and versions tested.
- Prompt and catalog construction.
- Whether runs were synthetic, human-reviewed, or production-derived.
- Randomness, repeat count, and confidence or variance where applicable.
- What the benchmark cannot conclude about real-world users.
Frequently asked questions
What is MCP tool-selection evaluation?
It is a test of whether an agent chooses the right MCP tool for an intent and then produces valid arguments, receives a useful result, recovers from errors, and completes the workflow safely.
Is tool-selection accuracy enough?
No. Selection, argument validity, execution success, recovery, completion, and safety should be measured separately because each reveals a different failure mode.
How many prompts do I need for an MCP benchmark?
There is no universal minimum. Use enough representative intents to cover your supported jobs, repeat runs where randomness matters, and publish the sample size and limitations instead of presenting a small fixture as a definitive benchmark.
See this on your own server
TrackMCP turns your MCP server's calls into adoption, workflows, and outcomes. One line to install.