LLM evaluation frameworks

line Orchestrator

The pipeline handles async execution, batching, caching, and threshold enforcement. It routes metrics efficiently and fails fast when critical gates are breached.

interface PipelineConfig {
  evaluators: Evaluator[];
  cache?: NodeCache;
  batchSize?: number;
  failFast?: boolean;
}

class EvalPipeline {
  private config: Required<PipelineConfig>;

  constructor(config: PipelineConfig) {
    this.config = {
      batchSize: config.batchSize ?? 10,
      failFast: config.failFast ?? false,
      cache: config.cache ?? new NodeCache({ stdTTL: 3600 }),
      evaluators: config.evaluators
    };
  }

  async run(contexts: EvaluationContext[]): Promise<Record<string, MetricResult[]>> {
    const results: Record<string, MetricResult[]> = {};
    
    for (let i = 0; i < contexts.length; i += this.config.batchSize) {
      const batch = contexts.slice(i, i + this.config.batchSize);
      const batchPromises = batch.map(ctx => this.evaluateSingle(ctx));
      const batchResults = await Promise.all(batchPromises);

      batchResults.forEach((res, idx) => {
        const key = `${contexts[i + idx].prompt.slice(0, 50)}_${contexts[i + idx].metadata?.version ?? 'v1'}`;
        results[key] = res;
      });

      if (this.config.failFast) {
        const hasCriticalFailure = batchResults.flat().some(r => !r.passed && r.name.includes('critical'));
        if (hasCriticalFailure) throw new Error('Critical evaluation gate failed. Pipeline halted.');
      }
    }

    return results;
  }

  private async evaluateSingle(ctx: EvaluationContext): Promise<MetricResult[]> {
    const cacheKey = JSON.stringify(ctx);
    const cached = this.config.cache.get<MetricResult[]>(cacheKey);
    if (cached) return cached;

    const results = await Promise.all(
      this.config.evaluators.map(e => e.evaluate(ctx))
    );

    this.config.cache.set(cacheKey, results);
    return results;
  }
}

Architecture Decisions & Rationale

1. Modular Evaluator Pattern: Separates validation logic from execution. New metrics (e.g., toxicity, latency, token efficiency) can be added without touching the pipeline. This enables A/B testing of evaluation strategies.
2. Async Batching & Rate Limiting: LLM judges hit API limits quickly. Batching prevents thundering herd issues and allows integration with provider-specific concurrency controls.
3. Deterministic Cache First: Evaluation is idempotent for identical inputs. Caching eliminates redundant API calls, reducing cost by 40-60% in CI environments where prompts repeat across runs.
4. Fail-Fast Gating: Not all metrics carry equal weight. Critical gates (e.g., PII leakage, safety violations) halt execution immediately, preventing downstream propagation of invalid outputs.
5. Normalized 0-1 Scoring: Forces consistent aggregation. Raw outputs (regex matches, judge scores, statistical distances) are normalized before reporting, enabling apples-to-apples comparison across metric types.

Pitfall Guide

1. Treating LLM-as-a-Judge as Ground Truth LLM judges inherit the same biases, hallucination patterns, and instruction-following drift as the target model. They exhibit mode collapse on ambiguous prompts and reward stylistic polish over factual accuracy. Always calibrate judges against human-labeled subsets and use deterministic guards for objective criteria.

2. Ignoring Evaluator Prompt Drift Changing an evaluator's prompt invalidates historical scores. Teams frequently update judge prompts to "improve accuracy" without versioning, causing regression in tracked metrics. Lock evaluator prompts, version them in source control, and run shadow evaluations before promoting changes.

3. Static Test Datasets Fixed evaluation sets lead to data leakage and overfitting. Models optimize for known prompts during fine-tuning or prompt iteration. Rotate test cases, inject adversarial variations, and use synthetic data generation to expand edge-case coverage. Maintain a held-out production sample that never enters the eval loop.

4. Metric Normalization Blindness Comparing raw scores across different metric types creates false confidence. A regex match rate of 0.85 and a judge score of 0.85 are not equivalent. Normalize all metrics to 0-1, apply weighting based on business impact, and track distributions rather than single-point aggregates.

5. Skipping Cost & Latency Budgets Evaluation pipelines often become slower than inference. Running 50 LLM judges per prompt at production scale burns budget and blocks deployments. Budget evaluations like you budget inference: set per-prompt cost caps, route cheap checks first, and reserve LLM judges for high-uncertainty cases.

6. No Version Control for Evaluations Evaluations are code. Without version control, teams cannot correlate metric shifts with model updates, prompt changes, or data pipeline modifications. Store eval configs, prompts, and thresholds alongside application code. Tag evaluation runs with commit SHAs for traceability.

7. Over-Indexing on Aggregate Scores Averages mask failure modes. A 0.92 overall score can hide 100% failure on critical edge cases. Track percentile distributions (p50, p90, p99), monitor failure clusters by category, and enforce minimum thresholds per segment rather than relying on global averages.

Production Bundle

Action Checklist

• Define evaluation contract: Establish strict TypeScript interfaces for contexts, metrics, and evaluator execution to enforce type safety and consistent aggregation.
• Implement deterministic guards first: Deploy schema validation, regex checks, and token/latency limits before introducing LLM judges to reduce cost and latency.
• Version all evaluator prompts: Treat judge prompts as production code. Store them in version control, tag evaluation runs, and run shadow comparisons before promotion.
• Set cost and latency budgets: Cap API calls per evaluation run, batch requests, cache identical inputs, and route cheap checks before expensive semantic validation.
• Normalize and weight metrics: Convert all scores to 0-1, apply business-impact weights, and track percentile distributions instead of relying on aggregate averages.
• Integrate with CI/CD gates: Block deployments on critical threshold breaches, allow warnings for non-critical metrics, and generate diff reports between runs.
• Rotate test datasets: Prevent overfitting by injecting adversarial variations, using synthetic edge-case generation, and maintaining a held-out production sample.

Decision Matrix

Configuration Template

import { EvalPipeline, DeterministicEvaluator, LLMJudgeEvaluator } from './eval-framework';
import NodeCache from 'node-cache';
import { z } from 'zod';
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const responseSchema = z.object({
  answer: z.string().min(1),
  confidence: z.number().min(0).max(1),
  sources: z.array(z.string()).optional()
});

const evaluators = [
  new DeterministicEvaluator('json-schema', responseSchema),
  new DeterministicEvaluator('pii-detector', /(?:\b\d{3}[-.]?\d{2}[-.]?\d{4}\b|\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b)/i),
  new LLMJudgeEvaluator(
    openai,
    'gpt-4o-mini',
    0.75,
    `Rate factual grounding 0-1. Output JSON: {"score": number, "reason": string}`
  ),
  new LLMJudgeEvaluator(
    openai,
    'gpt-4o-mini',
    0.80,
    `Check tone alignment with professional support guidelines. Output JSON: {"score": number, "reason": string}`
  )
];

const pipeline = new EvalPipeline({
  evaluators,
  cache: new NodeCache({ stdTTL: 1800, checkperiod: 300 }),
  batchSize: 20,
  failFast: true,
  thresholds: {
    'json-schema': 1.0,
    'pii-detector': 1.0,
    'llm-judge-factual': 0.75,
    'llm-judge-tone': 0.80
  }
});

export { pipeline, evaluators };

Quick Start Guide

1. Install dependencies: npm install zod node-cache openai typescript @types/node
2. Create config file: Copy the Configuration Template into eval.config.ts. Replace process.env.OPENAI_API_KEY with your provider key or swap the client for your preferred LLM API.
3. Run evaluation:

   import { pipeline } from './eval.config';
   const contexts = [{ prompt: 'What is the capital of France?', response: '{"answer":"Paris","confidence":0.95}', groundTruth: 'Paris is the capital.' }];
   const results = await pipeline.run(contexts);
   console.log(JSON.stringify(results, null, 2));
   

4. Integrate with CI: Add a GitHub Actions step that runs the pipeline on pull requests, fails on critical threshold breaches, and posts a metric diff comment to the PR using the pipeline's built-in reporting utilities.