hfulness, answer_relevance, context_precision from datasets import Dataset

Production-ready dataset structure

eval_data = { "question": ["What is the latency SLA for vector search?"], "answer": ["The SLA is <50ms for P95 queries on GPU-accelerated indices."], "contexts": [["Vector search engines optimize HNSW or IVF-PQ for sub-50ms P95 latency on A100 instances."]], "ground_truth": ["<50ms P95 latency on GPU-accelerated vector indices."] }

dataset = Dataset.from_dict(eval_data)

Metric configuration with LLM judge calibration

result = evaluate( dataset, metrics=[faithfulness, answer_relevance, context_precision], llm=OpenAIChat(model="gpt-4o-mini"), # Calibrated judge embeddings=OpenAIEmbeddings(model="text-embedding-3-small") )

Extract production metrics

print(result.to_pandas())

### Step 4: Architecture Decisions
1. **Judge Calibration**: Never use raw LLM judges. Apply temperature=0.1, system prompt anchoring, and few-shot examples from your domain. Cache judge outputs to avoid redundant API calls.
2. **Parallel Execution**: Use `asyncio` or Ray for batch evaluation. Metric computation is embarrassingly parallel; shard by query hash.
3. **Metric Weighting**: Implement domain-specific weighting. Financial compliance prioritizes faithfulness (0.6) over answer relevance (0.4). Technical support weights context precision higher.
4. **CI/CD Gating**: Fail deployments if weighted score drops >5% from baseline or if hallucination rate exceeds 2%.
5. **Observability Integration**: Ship metrics to Prometheus/Grafana or OpenTelemetry. Track distribution shifts using KS tests on metric histograms.

## Pitfall Guide

1. **Treating semantic similarity as factual correctness**  
   BERTScore measures distributional overlap, not logical entailment. High similarity scores mask citation fabrication. Mitigation: Always pair with faithfulness and context precision.

2. **Over-relying on uncalibrated LLM-as-judge**  
   Raw prompt judges exhibit prompt-sensitivity and domain bias. Mitigation: Anchor system prompts, use temperature=0.1, validate against human-labeled subsets quarterly.

3. **Ignoring context precision/recall trade-offs**  
   Optimizing recall floods the context window, degrading faithfulness. Optimizing precision drops critical chunks. Mitigation: Monitor both; use hybrid search (dense + sparse) to balance coverage and ranking quality.

4. **Single-metric optimization (Goodhart's law)**  
   Maximizing one metric collapses system robustness. Mitigation: Implement weighted composite scoring with domain-specific thresholds. Track metric correlation matrices.

5. **Static ground truth in dynamic domains**  
   Regulatory, medical, and technical documentation change quarterly. Stale ground truth inflates faithfulness artificially. Mitigation: Version ground-truth datasets; trigger re-evaluation on document updates.

6. **Skipping evaluation latency/cost tracking**  
   Evaluation pipelines can consume 30–40% of inference budget. Mitigation: Sample strategically, cache judge outputs, use lightweight models for pre-filtering before expensive faithfulness checks.

7. **Not validating metric-user satisfaction correlation**  
   High framework scores don't guarantee UX. Mitigation: Run monthly A/B tests correlating metric deltas with CSAT, resolution time, and escalation rates. Adjust weights accordingly.

## Production Bundle

### Action Checklist
- [ ] Define domain-specific metric weights (faithfulness, precision, relevance)
- [ ] Implement LLM judge calibration (temperature, anchoring, few-shot)
- [ ] Set up CI/CD gates with 5% degradation thresholds
- [ ] Instrument evaluation latency and cost tracking
- [ ] Version control ground-truth datasets with change triggers
- [ ] Establish online sampling strategy (5–10% stratified traffic)
- [ ] Run quarterly correlation analysis between metrics and CSAT

### Decision Matrix

| Framework | Metric Coverage | Eval Latency | Cost/10k | CI/CD Integration | Open Source |
|-----------|-----------------|--------------|----------|-------------------|-------------|
| RAGAS | Faithfulness, Relevance, Precision/Recall, Hallucination | Medium | $12–18 | GitHub Actions, GitLab CI | Yes |
| TruLens | Full suite + drift detection | High | $15–22 | Custom SDK, Airflow | Yes |
| DeepEval | Faithfulness, Answer Relevancy, Context Precision | Low-Medium | $8–14 | Pytest plugin, GitHub | Yes |
| Custom LLM-Judge | Flexible but unstandardized | Variable | $5–30 | Manual pipelines | Yes |

### Configuration Template
```yaml
# rag_eval_config.yaml
evaluation:
  mode: batch  # batch | online | hybrid
  sampling:
    strategy: stratified
    rate: 0.07
    dimensions: [query_complexity, retrieval_depth, domain]
  
  metrics:
    weights:
      faithfulness: 0.5
      context_precision: 0.3
      answer_relevance: 0.2
    thresholds:
      faithfulness_min: 0.80
      context_precision_min: 0.75
      hallucination_rate_max: 0.02

  judge:
    model: gpt-4o-mini
    temperature: 0.1
    cache_ttl_hours: 24
    calibration_samples: 50

  ci_cd:
    gate: true
    degradation_threshold: 0.05
    artifact_store: s3://eval-artifacts/
    alerting:
      channels: [slack, pagerduty]
      severity: warning

Quick Start Guide

1. Install dependencies: pip install ragas datasets openai pyyaml
2. Prepare evaluation dataset: Structure queries, responses, retrieved contexts, and ground truth into a Dataset object. Validate schema matches RAGAS requirements.
3. Configure judge & metrics: Load rag_eval_config.yaml, set LLM judge parameters, and define domain weights. Run a 100-sample dry run to validate latency and cost.
4. Execute & integrate: Run evaluate(), export results to pandas/CSV, and wire outputs to your CI/CD pipeline. Set up Prometheus metrics for real-time dashboarding.
5. Iterate: Monitor metric drift weekly. Adjust sampling rate and weights based on correlation with user feedback and domain updates.

RAG evaluation is not a validation checkpoint; it is a continuous observability layer. Systems that treat metrics as static scores fail in production. Systems that engineer evaluation as a weighted, calibrated, and integrated pipeline achieve measurable gains in grounding, precision, and user trust. The architecture decisions outlined here transform evaluation from an afterthought into a deployment gate.