ai-artifacts/
  ├── intents/
  │   ├── classify_support.txt
  │   └── route_to_agent.txt
  ├── generation/
  │   ├── summarize_thread.txt
  │   └── draft_response.txt
  └── evaluation/
      ├── judge_rubric.txt
      └── test_suite.json

interface TestCase {
  id: string;
  category: 'happy_path' | 'edge_case' | 'adversarial';
  input: Record<string, string>;
  assertions: AssertionRule[];
  semanticRubric?: string;
}

interface AssertionRule {
  type: 'json_schema' | 'regex' | 'length_bound' | 'field_presence';
  pattern: string;
  failMessage: string;
}

interface EvaluationSuite {
  promptFile: string;
  baselineVersion: number;
  cases: TestCase[];
}

ecutes the evaluation suite against the target model. 3. Compares results against the pinned baseline.

Here is a TypeScript evaluation runner that orchestrates both assertion and judge checks:

import { readFileSync } from 'fs';
import { z } from 'zod';
import { callLLM, evaluateWithJudge } from './llm-client';

async function runRegressionGate(suite: EvaluationSuite): Promise<GateResult> {
  const promptTemplate = readFileSync(suite.promptFile, 'utf-8');
  const results: TestOutcome[] = [];

  for (const testCase of suite.cases) {
    const renderedPrompt = interpolate(promptTemplate, testCase.input);
    const output = await callLLM(renderedPrompt, { temperature: 0.7 });

    const structuralPass = validateAssertions(output, testCase.assertions);
    
    let semanticPass = true;
    if (testCase.semanticRubric) {
      const baselineOutput = await fetchBaselineOutput(suite.promptFile, suite.baselineVersion, testCase.id);
      const judgeResult = await evaluateWithJudge({
        candidate: output,
        baseline: baselineOutput,
        rubric: testCase.semanticRubric,
        model: 'claude-haiku',
        temperature: 0
      });
      semanticPass = judgeResult.severity === 'no_regression';
    }

    results.push({
      caseId: testCase.id,
      structural: structuralPass,
      semantic: semanticPass,
      timestamp: Date.now()
    });
  }

  const gatePassed = results.every(r => r.structural && r.semantic);
  return { passed: gatePassed, details: results };
}

Architecture Decisions & Rationale

• Temperature Pinning for Judges: The judge model runs at temperature: 0. Semantic evaluation requires deterministic scoring. Introducing randomness into the evaluator creates flaky gates.
• Baseline Versioning: The pipeline compares against a specific version number, not the latest commit. This prevents moving-target comparisons and ensures reproducible diffs.
• Separation of Structural and Semantic Checks: Assertions run first. If a JSON schema fails, there is no point in paying for an LLM-judge call. This reduces cost and latency by ~40% on structured outputs.
• Registry Integration: Prompt registry services (including platforms offering free tiers of 3 prompts and 50 runs monthly) provide version history, diff visualization, and API-driven execution. They abstract away model routing and token counting.

Pitfall Guide

1. Unpinned Model Versions

Explanation: Both the target model and the judge model receive frequent updates. A minor patch can alter tokenization or reasoning patterns, causing previously passing tests to fail without prompt changes. Fix: Explicitly lock model_version in your evaluation configuration. Update versions deliberately during scheduled maintenance windows, not automatically.

2. Temperature-Induced Flakiness

Explanation: Running the target model at temperature: 0.7 or higher introduces output variance. A test may pass on one CI run and fail on the next, eroding trust in the gate. Fix: Use temperature: 0 for regression testing. If production requires higher temperature, run the test suite multiple times and apply a majority-vote or confidence-threshold policy.

3. Synthetic Test Overload

Explanation: Playground-generated inputs are clean, well-formatted, and lack production noise. Tests built on synthetic data rarely catch real-world failures. Fix: Sample inputs from production logs. Anonymize PII, deduplicate, and curate a representative distribution. Real inputs are worth 10x synthetic ones.

4. Judge Prompt Drift

Explanation: The evaluation rubric itself is a prompt. If the judge prompt changes between runs, scores become incomparable. Fix: Version the judge prompt alongside target prompts. Store it in ai-artifacts/evaluation/judge_rubric.txt and include it in your CI diff checks.

5. Ignoring Cost Accumulation

Explanation: LLM-judge calls cost fractions of a cent each, but running them across hundreds of test cases on every PR quickly drains budgets. Fix: Implement input sampling for non-critical prompts. Cache results for unchanged inputs. Set up cost alerts at 80% of your monthly threshold.

6. Treating Semantic Scores as Binary

Explanation: Judges return probabilities or severity levels, not strict booleans. Forcing a hard pass/fail on nuanced outputs creates false positives. Fix: Configure configurable thresholds. For example, fail only if severity === 'critical_regression' or score < 0.85. Log warnings for minor drift without blocking merges.

7. Missing Adversarial Coverage

Explanation: Prompts optimized for happy paths often break under injection, contradictory instructions, or out-of-distribution inputs. Fix: Mandate at least one adversarial test case per prompt. Include common jailbreak patterns, role-confusion attempts, and instruction-overload scenarios.

Production Bundle

Action Checklist

• Centralize all prompts in a version-controlled directory with plain text templates
• Curate 5-30 production-representative inputs per prompt, categorized by use case
• Implement structural assertions for JSON, regex, and schema validation
• Configure an LLM-judge at temperature 0 for semantic comparison against pinned baselines
• Lock target and judge model versions in CI configuration
• Add path filters to trigger pipelines only on prompt artifact changes
• Set cost thresholds and logging for judge evaluation runs
• Document rollback procedures and baseline versioning policies

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Structured data extraction (JSON, forms, payloads)	Rule-Based Assertions	Deterministic, instant, zero cost	$0.00
Freeform summaries, rewrites, tone adjustments	LLM-as-Judge	Handles semantic nuance and flexible correctness	$0.001-$0.01 per run
Mission-critical customer-facing flows	Hybrid Pipeline	Catches structural breaks and semantic drift simultaneously	$0.001-$0.005 per run
High-volume, low-risk internal prompts	Assertion-Only	Cost efficiency outweighs semantic precision needs	$0.00
Creative or marketing generation	LLM-as-Judge with Human Review	Semantic quality requires nuanced evaluation; gate with manual approval fallback	$0.005-$0.02 per run

Configuration Template

# .github/workflows/prompt-regression.yml
name: Prompt Regression Gate
on:
  pull_request:
    paths:
      - 'ai-artifacts/**/*.txt'

env:
  REGISTRY_API_KEY: ${{ secrets.PROMPT_REGISTRY_KEY }}
  JUDGE_MODEL: claude-haiku
  TARGET_TEMP: 0

jobs:
  validate-prompts:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Push prompts to registry
        run: |
          npm run registry:push -- --dir ai-artifacts --message "PR #${{ github.event.pull_request.number }}"

      - name: Execute evaluation suite
        run: npm run eval:run -- --suite ai-artifacts/evaluation/test_suite.json --baseline latest

      - name: Report results
        if: always()
        run: |
          cat eval-report.json | jq '.details[] | select(.structural == false or .semantic == false)'
          exit $(jq '.passed' eval-report.json)

Quick Start Guide

1. Initialize the artifact directory: Create ai-artifacts/ and move all active prompts into .txt files. Add placeholder syntax for dynamic inputs.
2. Install the evaluation CLI: Run npm install @your-org/prompt-eval and configure your registry API key in environment variables.
3. Generate baseline tests: Use npm run eval:init -- --prompt ai-artifacts/generation/summarize_thread.txt --count 10 to scaffold test cases from recent production logs.
4. Commit the workflow: Add the GitHub Actions YAML to .github/workflows/. Push a test prompt change to verify the gate triggers, executes assertions, runs the judge, and blocks merges on regression.
5. Monitor and iterate: Review CI logs for false positives. Adjust semantic thresholds, add adversarial cases, and lock model versions. Scale to 30 test cases for mission-critical prompts.

Prompt regression testing transforms LLM development from experimental iteration into engineering discipline. By versioning artifacts, separating structural and semantic validation, and enforcing CI gates, teams eliminate silent degradation, reduce rollback cycles, and maintain consistent output quality at scale. The infrastructure requires minimal setup, but the operational discipline it enforces pays compounding dividends as prompt complexity and model dependencies grow.