Beyond Model Scores: A Rigorous Framework for Evaluating AI Agent Tooling

Current Situation Analysis

The industry has spent years perfecting evaluation pipelines for foundational language models. Standard suites measure single-turn accuracy, instruction following, and reasoning depth. When teams pivot to evaluating agent-augmented tooling—specifically Model Context Protocol (MCP) servers for code intelligence—most simply port these existing frameworks over. This approach fails because the evaluation variable fundamentally shifts. You are no longer measuring model capability; you are measuring how a fixed model interacts with an external interface, how it routes queries, and how it synthesizes tool outputs into developer-ready answers.

The core misunderstanding lies in treating agent tooling as a passive knowledge source rather than an active behavioral modifier. When an MCP server is attached to an agent, it changes the token distribution, alters the exploration path, and introduces new failure modes like citation hallucination and tool-selection bias. Frameworks built around prompt → response → grade cannot capture multi-turn tool orchestration, citation grounding against a live filesystem, or the distinction between tool fluency and actual answer quality.

Empirical evidence from recent code-intelligence evaluations highlights the severity of this gap. When benchmarking multiple MCP servers against a fixed model (Opus 4.7 with 1M context), two out of five tested tools actually degraded agent performance compared to a zero-tool baseline using only standard shell utilities. Citation grounding—the percentage of file and line references that actually exist in the target repository—varied dramatically, ranging from 61.9% to 89.2%. These numbers prove that attaching a tool does not guarantee improvement. Without a methodology designed specifically for agent-tool dynamics, benchmarks reward verbosity and tool usage frequency rather than developer utility.

WOW Moment: Key Findings

The most critical insight from rigorous agent-tool benchmarking is that tool usage frequency and answer quality are orthogonal metrics. Collapsing them into a single headline score masks degradation and rewards friction. By decoupling fairness (actual answer improvement) from adoption (how fluently the agent uses the tool), a clear performance hierarchy emerges that standard frameworks miss entirely.

Evaluation Layer	Metric	Baseline (No MCP)	Top-Performing MCP	Underperforming MCP
Fairness	Answer Quality + Efficiency	77.2%	81.3%	74.9%
Adoption	Tool Fluency + Discoverability	0% (Structural)	88.4%	62.1%
Citation Grounding	Verified File/Line Existence	80.8%	89.2%	61.9%
Judge Stability	Scenario-Level Variance (σ)	0.014	0.011	0.018

This finding matters because it forces a architectural decision in evaluation design: if adoption metrics bleed into the primary score, the benchmark becomes a survey of tool integration rather than a measure of developer value. The data shows that two MCP servers scored below baseline on fairness despite high adoption rates. They increased token consumption and wall-clock time while introducing citation hallucinations that required manual verification. Separating these layers reveals which tools genuinely accelerate development and which merely add orchestration overhead.

Core Solution

Building a trustworthy evaluation harness for agent tooling requires abandoning single-turn paradigms and constructing a multi-layered verification pipeline. The architecture must isolate model behavior, pin the execution environment, partition transcript data, and route scoring through variance-aware filters.

Step 1: Environment Pinning & Citation Grounding

Agent answers must be verified against a deterministic filesystem state. Instead of grading text against text, extract all file and line references, then validate them against a specific commit hash.

interface CitationReference {
  filePath: string;
  lineNumber: number;
  symbolName?: string;
}

interface GroundingResult {
  status: 'grounded' | 'unresolved' | 'hallucinated';
  details: string;
}

class FilesystemVerifier {
  constructor(private repoPath: string, private commitSha: string) {}

  async verifyCitation(ref: CitationReference): Promise<GroundingResult> {
    const fullPath = path.join(this.repoPath, ref.filePath);
    const exists = await fs.pathExists(fullPath);
    
    if (!exists) {
      return { status: 'unresolved', details: 'File path not found in pinned commit' };
    }

    const content = await fs.readFile(fullPath, 'utf-8');
    const lineCount = content.split('\n').length;

    if (ref.lineNumber > lineCount) {
      return { status: 'hallucinated', details: `Line ${ref.lineNumber} exceeds EOF (${lineCount})` };
    }

    if (ref.symbolName) {
      const symbolLine = await this.resolveSymbol(fullPath, ref.symbolName);
      const offset = Math.abs(symbolLine - ref.lineNumber);
      if (offset > 5) {
        return { status: 'hallucinated', details: `Symbol offset exceeds ±5 line tolerance` };
      }
    }

    return { status: 'grounded', details: 'Reference verified within bounds' };
  }
}

Rationale: Hallucinated citations are the strongest signal of tool-induced degradation. A file existing but containing a fabricated line number indicates the agent is inventing context rather than reading it. The ±5 line tolerance for symbol resolution accounts for AST parsing drift without penalizing minor formatting differences.

Step 2: Transcript Partitioning

Agent transcripts contain assistant prose, tool invocations, and tool outputs. Scoring must only evaluate the prose. Tool inputs frequently contain keyword matches that artificially inflate relevance scores.

interface TranscriptSegment {
  role: 'assistant' | 'tool_call' | 'tool_result';
  content: string;
}

class TranscriptPartitioner {
  partition(rawTranscript: TranscriptSegment[]) {
    const answerText: string[] = [];
    const auditText: string[] = [];

    for (const segment of rawTranscript) {
      if (segment.role === 'assistant') {
        answerText.push(segment.content);
      } else {
        auditText.push(segment.content);
      }
    }

    return {
      answerText: answerText.join('\n'),
      auditText: auditText.join('\n')
    };
  }
}

Rationale: Keyword matching against tool calls creates a structural bias toward grep-heavy approaches. Semantic search tools that avoid echoing symbol names in tool inputs would be unfairly penalized. Partitioning ensures scoring measures developer-facing output, not internal routing mechanics.

Step 3: Dual-Layer Scoring Architecture

Fairness and adoption must be computed independently. Fairness measures whether the developer received a better, faster, cheaper answer. Adoption measures how effectively the agent utilized the attached tooling.

interface ScoringWeights {
  fairness: { keywordCoverage: number; llmQuality: number; citationGrounding: number; efficiency: number };
  adoption: { toolFluency: number; discoverability: number };
}

class DualLayerScorer {
  constructor(private weights: ScoringWeights) {}

  computeFairness(metrics: Record<string, number>): number {
    return (
      metrics.keywordCoverage * this.weights.fairness.keywordCoverage +
      metrics.llmQuality * this.weights.fairness.llmQuality +
      metrics.citationGrounding * this.weights.fairness.citationGrounding +
      metrics.efficiency * this.weights.fairness.efficiency
    );
  }

  computeAdoption(metrics: Record<string, number>): number {
    return (
      metrics.toolFluency * this.weights.adoption.toolFluency +
      metrics.discoverability * this.weights.adoption.discoverability
    );
  }
}

Rationale: Folding adoption into fairness creates a self-fulfilling benchmark where tools are rewarded for being called, regardless of whether they improve outcomes. The baseline (no MCP) receives a structural zero on adoption by design. Keeping layers separate allows tool-vs-tool comparison on adoption while preserving a clean fairness metric that baseline can legitimately win.

Step 4: Variance-Aware Judge Routing

LLM-as-judge systems exhibit stochastic behavior. Criterion-level scores fluctuate more than aggregated scenario scores. Routing decisions must account for this noise floor.

class JudgeVarianceTracker {
  private runs: Map<string, number[]> = new Map();

  recordScore(scenarioId: string, layer: 'criterion' | 'step' | 'scenario', score: number) {
    const key = `${scenarioId}:${layer}`;
    if (!this.runs.has(key)) this.runs.set(key, []);
    this.runs.get(key)!.push(score);
  }

  getStability(scenarioId: string, layer: 'criterion' | 'step' | 'scenario'): number {
    const key = `${scenarioId}:${layer}`;
    const scores = this.runs.get(key) || [];
    if (scores.length < 2) return 0;
    const mean = scores.reduce((a, b) => a + b, 0) / scores.length;
    const variance = scores.reduce((acc, val) => acc + Math.pow(val - mean, 2), 0) / scores.length;
    return Math.sqrt(variance);
  }
}

Rationale: Criterion-level standard deviation often exceeds 0.07, crossing into unreliable territory. Step-level aggregation drops it below 0.05, and scenario-level means stabilize near 0.014. Ranking must only use scenario-level outputs. Criterion rationales should be treated as diagnostic commentary, not decision data.

Step 5: Anti-Drift Anchoring

Rubric tuning inevitably drifts toward optimizing the metric rather than measuring capability. A frozen held-out set with cryptographic locking prevents Goodhart's law from corrupting the evaluation loop.

interface AnchorLock {
  scenarioId: string;
  transcriptHash: string;
  rubricHash: string;
  goldScoreHash: string;
}

class DriftPreventionGuard {
  async validateAnchor(lock: AnchorLock): Promise<boolean> {
    const currentTranscript = await this.computeHash(lock.scenarioId + '.json');
    const currentRubric = await this.computeHash('rubric.yaml');
    const currentGold = await this.computeHash('gold_scores.json');

    const integrity = 
      currentTranscript === lock.transcriptHash &&
      currentRubric === lock.rubricHash &&
      currentGold === lock.goldScoreHash;

    if (!integrity) return false;

    const spearman = await this.computeSpearmanCorrelation();
    return spearman >= 0.85;
  }
}

Rationale: The evaluation loop must refuse to proceed if any component hash drifts or if Spearman correlation against hand-graded gold scores drops below 0.85. This creates a hard boundary between rubric refinement and metric gaming.

Pitfall Guide

1. Scoring Tool Inputs as Answers

Explanation: Keyword matchers that scan the entire transcript count tool invocations as evidence of relevance. An agent calling Grep("TopicCreator") registers a hit even if the tool returns empty results. Fix: Strictly partition transcripts. Route only assistant prose to the scoring engine. Keep tool I/O in a separate audit stream for diagnostics only.

2. Collapsing Adoption into Fairness

Explanation: Measuring how often an agent calls a tool and adding it to the primary quality score rewards friction. Tools that require multiple round-trips to resolve simple queries appear superior to tools that answer directly. Fix: Maintain separate scoring namespaces. Fairness measures developer outcome; adoption measures integration fluency. Never merge them in the headline metric.

3. Ignoring LLM Judge Variance at the Criterion Level

Explanation: Treating a 0.04 delta in a single criterion (e.g., "uncertainty handling") as statistically significant leads to false rankings. Criterion-level standard deviation frequently exceeds 0.07. Fix: Aggregate scores to the step or scenario level before ranking. Use criterion breakdowns for qualitative analysis, not quantitative gating.

4. Naive Citation Regex Without Bounds Checking

Explanation: Extracting file:line patterns without verifying file existence or line ranges produces false positives. Agents frequently cite valid filenames with fabricated line numbers. Fix: Implement a three-bucket verification system: grounded (file exists, line in range, symbol within ±5 lines), unresolved (path missing), hallucinated (file exists but line exceeds EOF or symbol offset is too large).

5. Rubric Drift Without Held-Out Anchors

Explanation: Iteratively tuning scoring weights to improve benchmark performance eventually optimizes the rubric instead of measuring capability. Scores inflate while real-world utility stagnates. Fix: Freeze three held-out scenarios with SHA256-locked transcripts and rubrics. Require Spearman correlation > 0.85 against hand-graded gold scores before accepting any rubric change.

6. Assuming Tool Usage Equals Value

Explanation: High adoption scores indicate the agent knows how to call the tool, not that the tool improves the answer. Two tested MCP servers scored below baseline on fairness despite strong adoption metrics. Fix: Treat adoption as a secondary diagnostic layer. Primary ranking must rely on fairness metrics that baseline can legitimately compete for.

7. Benchmarking Cost Without Contextualizing Token Routing

Explanation: Reporting raw token counts without distinguishing between reasoning tokens, tool I/O tokens, and citation verification tokens obscures efficiency bottlenecks. Fix: Tag token usage by phase. Separate agent reasoning costs from tool orchestration overhead. Report efficiency as a weighted composite of time, tokens, and answer completeness.

Production Bundle

Action Checklist

• Pin repository state: Lock all evaluation scenarios to specific commit SHAs to ensure deterministic filesystem verification.
• Partition transcripts: Implement strict separation between assistant prose and tool I/O before any scoring occurs.
• Build citation verifier: Create a three-bucket grounding system (grounded/unresolved/hallucinated) that validates paths, line ranges, and symbol offsets.
• Decouple scoring layers: Compute fairness and adoption independently. Never fold adoption metrics into the primary quality score.
• Characterize judge variance: Run duplicate judge passes on 12+ transcripts. Only use scenario-level aggregates for ranking; discard criterion-level deltas under 0.05.
• Lock held-out anchors: Freeze three scenarios with SHA256 hashes for transcripts, rubrics, and gold scores. Enforce Spearman correlation ≥ 0.85.
• Tag token phases: Instrument the harness to separate reasoning tokens from tool I/O and verification overhead for accurate efficiency reporting.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Evaluating code-intelligence MCP servers	Dual-layer scoring with strict citation grounding	Separates tool fluency from actual developer utility; catches hallucination drift	+15% runtime for filesystem verification
Benchmarking general-purpose agent tools	LLM-judge focused with step-level aggregation	Tool outputs are less filesystem-dependent; scenario-level variance is stable	Baseline API cost (~$40 per full run)
High-frequency iteration cycles	Hold-out anchor with Spearman gating	Prevents rubric gaming during rapid tuning; maintains correlation with human judgment	Negligible compute, requires manual gold-score creation
Resource-constrained environments	Criterion-level filtering disabled, scenario-only ranking	Reduces judge calls by 60%; avoids noise-floor decisions	-40% token cost, slightly lower diagnostic granularity

Configuration Template

// bench.config.ts
export const BenchmarkConfig = {
  model: {
    provider: 'anthropic',
    name: 'opus-4.7',
    contextWindow: 1_000_000,
    maxTokens: 8192
  },
  scoring: {
    fairness: {
      keywordCoverage: 0.10,
      llmQuality: 0.55,
      citationGrounding: 0.15,
      efficiency: 0.20
    },
    adoption: {
      toolFluency: 0.60,
      discoverability: 0.40
    }
  },
  verification: {
    citation: {
      symbolOffsetTolerance: 5,
      hallucinationThreshold: 'exceeds_eof'
    },
    judge: {
      varianceFloor: 0.05,
      rankingLayer: 'scenario',
      duplicateRuns: 2
    }
  },
  anchors: {
    heldOutCount: 3,
    spearmanThreshold: 0.85,
    lockfile: 'bench/locked/held-out.lock'
  },
  execution: {
    wallClockLimit: 1200, // seconds
    maxConcurrentSessions: 5,
    outputDir: 'bench/results'
  }
};

Quick Start Guide

1. Initialize the harness: Clone the evaluation repository and install dependencies. Run npm run bench:init to generate the directory structure and lockfile templates.
2. Configure your tool: Create a shell wrapper in bench/tools/yourtool.sh that starts your MCP server binary with the required flags. Make it executable.
3. Run a single cell: Execute npm run bench:cell --tool yourtool --repo target_repo. The harness will spin up the agent, attach the tool, run the scripted scenario, and output scored results to bench/results/.
4. Validate anchors: Run npm run bench:verify to check SHA256 integrity of held-out scenarios and compute Spearman correlation against gold scores. The pipeline will halt if correlation drops below 0.85.
5. Generate report: Execute npm run bench:report to compile fairness and adoption metrics across all tools. Review citation grounding breakdowns and judge variance logs before drawing conclusions.