mcp-eval

​

Why teams run it

• Catch regressions when prompts, workflows, or model settings change
• Confirm that the right MCP tools fire in the expected order with the expected payloads
• Exercise recovery paths such as human-input pauses or fallback workflows
• Produce repeatable evidence—reports, traces, and badges—that a release is safe

​

What you can cover

​

Install mcp-eval

uv tool install mcpevals      # CLI
uv add mcpevals               # project dependency
mcp-eval init                 # scaffold config, tests/, and datasets/

pip install mcpevals
mcp-eval init

​

Register what you test

1. Register servers with the same command or endpoint your agent uses:
   mcp-eval server add \
     --name fetch \
     --transport stdio \
     --command "uv" "run" "python" "-m" "mcp_servers.fetch"
   
2. Register agents by pointing to an AgentSpec, an instantiated Agent, or your MCPApp:
   # tests/config/targets.yaml
   agents:
     - name: finder
       type: agent_spec
       path: ../../examples/basic/mcp_basic_agent/mcp_agent/agents/finder.py
   servers:
     - name: fetch
       transport: stdio
       command: ["uv", "run", "python", "-m", "mcp_servers.fetch"]
   
3. When you introduce a new workflow or capability, run mcp-eval generate to draft scenario ideas with LLM assistance.

​

Structure evaluations

​

Decorator tasks

from mcp_eval import Expect, task

@task("Finder summarizes Example Domain")
async def test_finder_fetch(agent, session):
    response = await agent.generate_str("Fetch https://example.com and summarize it.")

    await session.assert_that(Expect.tools.was_called("fetch"))
    await session.assert_that(Expect.content.contains("Example Domain"), response=response)
    await session.assert_that(Expect.performance.max_iterations(3))

​

Pytest suites

import pytest
from mcp_eval import create_agent, Expect

@pytest.mark.asyncio
async def test_finder_fetch_pytest():
    agent = await create_agent("finder")
    response = await agent.generate_str("Fetch https://example.com")
    assert "Example Domain" in response
    await Expect.tools.was_called("fetch").evaluate(agent.session)

​

Dataset runs

from mcp_eval import Case, Dataset, Expect
from mcp_eval import create_agent

dataset = Dataset(
    cases=[
        Case(
            name="fetch_example_domain",
            inputs="Fetch https://example.com and summarize it.",
            evaluators=[Expect.tools.was_called("fetch")],
        )
    ]
)

async def run_case(prompt: str) -> str:
    agent = await create_agent("finder")
    return await agent.generate_str(prompt)

report = await dataset.evaluate(run_case)  # call from an async test or helper

​

Run and inspect

mcp-eval run tests/   # decorator, dataset, and CLI suites
uv run pytest -q tests

metrics = session.get_metrics()
span_tree = session.get_span_tree()

​

Pick a focus area

• Work through end-to-end agent scenarios in Agent Evaluation.
• Validate server behavior and tool contracts in MCP Server Evaluation.
• Refer back to mcp-eval.ai for extended guides, configuration options, and community examples.

​

Observability, reports, and CI/CD

• OpenTelemetry traces flow to Grafana, Honeycomb, Pydantic Logfire, or any OTEL target
• JSON/Markdown/HTML reports are ready for CI artifacts or release notes
• Reusable GitHub Actions (mcp-eval/.github/actions/mcp-eval/run) publish test results, summaries, and badges