uires non-empty failure_details")

```python
# tests/unit/test_artifact_contracts.py
import pytest
from pathlib import Path
from src.validation.artifact_schemas import DiagnosticReport

@pytest.mark.parametrize("fixture_file", ["success_case.json", "failure_case.json"])
def test_diagnostic_artifact_schema(fixture_file: str) -> None:
    fixture_path = Path("fixtures/diagnostic_stage") / fixture_file
    raw_data = fixture_path.read_text()
    
    # Pydantic raises ValidationError on schema mismatch
    report = DiagnosticReport.model_validate_json(raw_data)
    assert report.confidence_score >= 0.0
    assert isinstance(report.supporting_evidence, list)

Architecture Rationale: Fixtures replace live inference. By saving real production outputs as test data, you capture the exact shape of successful and failed runs. Schema validation runs in milliseconds, requires zero API calls, and prevents downstream stages from crashing on malformed payloads.

Layer 2: Connectivity & Routing Validation (Phase-Level Integration)

Integration testing in LLM pipelines focuses on two questions: Does stage N's output satisfy stage N+1's input requirements? Does the routing engine correctly branch based on confidence thresholds, retry limits, or failure states?

Routing logic must be extracted into pure functions. This eliminates non-determinism from integration tests and allows exhaustive edge-case coverage.

# src/pipeline/routing_engine.py
from typing import Literal

RoutingDecision = Literal["advance", "gate_review", "retry_stage", "escalate_human"]

class StageRouter:
    def __init__(self, max_retries: int = 3, high_confidence: float = 0.95, low_confidence: float = 0.60):
        self.max_retries = max_retries
        self.high_confidence = high_confidence
        self.low_confidence = low_confidence

    def evaluate(self, confidence: float, passed: bool, retry_count: int) -> RoutingDecision:
        if passed and confidence >= self.high_confidence:
            return "advance"
        if passed and confidence >= self.low_confidence:
            return "gate_review"
        if not passed and retry_count < self.max_retries:
            return "retry_stage"
        return "escalate_human"

# tests/integration/test_routing_logic.py
import pytest
from src.pipeline.routing_engine import StageRouter

@pytest.fixture
def router():
    return StageRouter(max_retries=3, high_confidence=0.95, low_confidence=0.60)

def test_routing_boundaries(router: StageRouter) -> None:
    assert router.evaluate(confidence=0.98, passed=True, retry_count=0) == "advance"
    assert router.evaluate(confidence=0.75, passed=True, retry_count=0) == "gate_review"
    assert router.evaluate(confidence=0.45, passed=False, retry_count=1) == "retry_stage"
    assert router.evaluate(confidence=0.45, passed=False, retry_count=3) == "escalate_human"

Architecture Rationale: Pure routing functions run deterministically. They can be tested against every mathematical boundary condition in milliseconds. Data flow validation complements this by asserting that output schemas contain all fields required by downstream context windows.

Layer 3: System Regression & Metric Baselines (Workflow-Level Validation)

End-to-end tests execute the full pipeline against curated scenarios. These tests are expensive and slow, so they run only during release candidates or major architectural changes. Each scenario defines expected execution paths and acceptable metric ranges.

# regression/scenarios.yml
test_suites:
  - id: SYS-E2E-001
    label: "Standard resolution path"
    inputs:
      ticket_id: "PROJ-1042"
      description: "NullReferenceException in config parser"
    expected_sequence:
      - ingestion: completed
      - analysis: completed
      - diagnosis: completed
      - remediation: completed
      - verification: completed
    metric_thresholds:
      full_automation_rate: 0.70
      remediation_attempts: 1.5
      human_intervention_rate: 0.20

Metric baselines must be tracked across runs. Deviations indicate model degradation, prompt drift, or infrastructure changes.

# src/evaluation/metric_collector.py
from dataclasses import dataclass, field
from typing import Dict

@dataclass
class PipelineMetrics:
    automation_success_rate: float = 0.0
    avg_remediation_rounds: float = 0.0
    parallel_candidate_pass_rate: float = 0.0
    human_gate_trigger_rate: float = 0.0
    violations: Dict[str, float] = field(default_factory=dict)

    def validate_against_baseline(self, baseline: "PipelineMetrics") -> bool:
        checks = [
            self.automation_success_rate >= baseline.automation_success_rate,
            self.avg_remediation_rounds <= baseline.avg_remediation_rounds,
            self.parallel_candidate_pass_rate >= baseline.parallel_candidate_pass_rate,
            self.human_gate_trigger_rate <= baseline.human_gate_trigger_rate
        ]
        return all(checks)

Architecture Rationale: E2E tests are regression gates, not development feedback loops. They compare current runs against historical baselines. Threshold violations block deployments until root causes are identified.

Distributed Tracing Integration

Observability transforms opaque pipeline runs into queryable execution graphs. Langfuse provides trace, span, and event primitives that map directly to workflow architecture.

# src/observability/pipeline_instrumentor.py
from langfuse import Langfuse
from typing import Any, Dict

class PipelineTracer:
    def __init__(self, public_key: str, secret_key: str, host: str):
        self.client = Langfuse(public_key=public_key, secret_key=secret_key, host=host)

    def start_workflow_trace(self, ticket_id: str, version: str) -> Any:
        return self.client.trace(
            name=f"pipeline-execution:{ticket_id}",
            input={"ticket_id": ticket_id},
            metadata={"pipeline_version": version}
        )

    def record_stage_span(self, trace: Any, stage_name: str, context: Dict[str, Any], result: Dict[str, Any]) -> None:
        span = trace.span(
            name=stage_name,
            input=context,
            output=result,
            level="WARNING" if not result.get("success") else "DEFAULT"
        )
        span.end()

    def log_intervention_event(self, trace: Any, trigger_reason: str, confidence_value: float) -> None:
        trace.event(
            name="human_review_triggered",
            metadata={"reason": trigger_reason, "confidence_at_trigger": confidence_value}
        )

Architecture Rationale: Traces capture execution order, latency, token consumption, and error states. Spans isolate stage performance. Events mark business-critical transitions like human escalations. This structure eliminates log diving and enables aggregate analysis across thousands of runs.

Pitfall Guide

Pitfall Name	Explanation	Production Fix
Schema Drift Ignorance	LLM outputs gradually change shape across model versions or prompt tweaks. Tests pass until a downstream stage crashes on a missing field.	Implement schema versioning. Store schema_version in every artifact. Add CI checks that reject payloads with mismatched versions.
Over-Mocking Inference	Replacing LLM calls with static mocks hides latency, token budget, and formatting inconsistencies that only appear in production.	Use real historical outputs as fixtures. Reserve mocking only for rate-limit simulation or network failure testing.
Hardcoded Routing Thresholds	Confidence cutoffs (e.g., 0.95) are treated as constants. Model updates shift output distributions, causing unexpected gate triggers.	Externalize thresholds to configuration files. Implement dynamic calibration using rolling window statistics from trace data.
Tracing Noise vs. Signal	Logging every intermediate variable creates massive trace payloads, increasing storage costs and obscuring root causes.	Trace only stage boundaries, routing decisions, and error states. Use sampling for high-frequency internal steps.
Baseline Staleness	Metric thresholds are set once and never updated. As pipeline quality improves, old baselines become meaningless or trigger false positives.	Automate baseline regeneration quarterly. Use statistical process control (SPC) charts to detect gradual metric drift.
Ignoring Token/Latency Budgets	Tests validate correctness but ignore cost. A pipeline that passes all assertions may still exceed budget due to verbose prompts or retry loops.	Add cost-aware assertions to Layer 3. Track tokens_per_stage and latency_p95. Fail CI if budgets exceed 10% of baseline.
Coupling Test Data to Environment	Fixtures contain absolute paths, environment-specific IDs, or hardcoded timestamps. Tests break when moved between CI runners.	Normalize fixtures using relative paths and synthetic identifiers. Strip environment-specific metadata before serialization.

Production Bundle

Action Checklist

• Define explicit output schemas for every pipeline stage using a validation library
• Capture production artifacts as test fixtures covering success, failure, and edge cases
• Extract routing logic into pure functions with exhaustive boundary testing
• Validate data handoffs by asserting downstream context requirements match upstream outputs
• Establish metric baselines for automation rate, retry rounds, candidate pass rate, and gate triggers
• Instrument every execution with traces, spans, and business events using a dedicated observability SDK
• Implement baseline comparison logic that blocks deployments on threshold violations
• Externalize routing thresholds and metric limits to configuration files for dynamic tuning

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Rapid iteration on prompt templates	Layer 1 schema validation + Layer 2 routing tests	Catches structural breaks without inference costs	Near-zero API spend
Model version upgrade or provider migration	Layer 3 regression suite + trace comparison	Detects distribution shifts, latency changes, and metric degradation	Moderate API spend, high ROI on stability
New stage integration or data format change	Layer 2 connectivity tests + fixture validation	Ensures downstream consumers receive required fields before full pipeline execution	Low cost, prevents cascading failures
Production incident investigation	Langfuse trace query + span aggregation	Isolates failing stage, token usage, and routing decisions without log reconstruction	Zero additional cost, reduces MTTR
Budget optimization or scaling planning	Trace analytics + metric baseline tracking	Identifies high-cost stages, retry loops, and gate trigger patterns	Enables targeted prompt engineering or caching

Configuration Template

# config/pipeline_validation.yml
schema_registry:
  version: "2.1.0"
  strict_mode: true
  auto_reject_mismatched: true

routing_engine:
  max_retries: 3
  thresholds:
    high_confidence: 0.95
    low_confidence: 0.60
    dynamic_calibration_window: 100_runs

metric_baselines:
  automation_success_rate: 0.72
  avg_remediation_rounds: 1.4
  parallel_candidate_pass_rate: 0.83
  human_gate_trigger_rate: 0.18
  violation_tolerance: 0.05

observability:
  provider: "langfuse"
  trace_sampling_rate: 1.0
  span_level: "stage_boundary"
  event_capture: ["gate_trigger", "escalation", "timeout"]

Quick Start Guide

1. Initialize schema contracts: Create Pydantic models for each stage's output. Save two production artifacts per stage (success/failure) as JSON fixtures.
2. Build routing tests: Extract branching logic into pure functions. Write parameterized tests covering high/medium/low confidence, retry exhaustion, and failure states.
3. Configure regression scenarios: Define YAML test cases with expected execution sequences and metric thresholds. Implement baseline comparison logic that outputs delta reports.
4. Instrument execution: Wrap pipeline entry points with trace initialization. Record spans at stage boundaries and events at human interventions. Connect to Langfuse or equivalent observability platform.
5. Integrate with CI: Run Layer 1 and Layer 2 tests on every commit. Trigger Layer 3 regression suite on release branches. Block merges if metric deltas exceed configured tolerance.