User reports "bad answer"
        ↓
Check context_recall
        ├─ Low ────→ Retrieval problem
        │             ├─ Key content not retrieved
        │             ├─ Chunks too small or too large
        │             └─ Top-K too low
        │
        └─ Normal ──→ Check faithfulness
                          ├─ Low ────→ Generation problem (hallucination)
                          │             └─ Prompt encourages model to go beyond context
                          │
                          └─ Normal ──→ Check answer_relevancy
                                            └─ Low ──→ Off-topic answer
                                                       └─ Prompt force

s rigid structure

### Failure Mode 1: Low Retrieval Recall

p1_pipeline = RAGPipeline( chunk_size=64, # Extremely small — docs shattered into fragments chunk_overlap=0, top_k=1, # Only 1 chunk retrieved — most info is lost prompt_type="baseline", )

**Root Cause**: Documents are fragmented into 64-character tokens. Concepts are distributed across multiple chunks. `top_k=1` retrieves only one fragment, leaving the LLM with incomplete semantics. `context_recall` drops sharply as key content remains unretrieved.

### Failure Mode 2: Hallucination

p2_pipeline = RAGPipeline( chunk_size=512, chunk_overlap=50, top_k=4, prompt_type="hallucination", # Hallucination-inducing prompt )

PROMPT_HALLUCINATION = ChatPromptTemplate.from_messages([ ("system", "You are an encyclopedic AI assistant with broad knowledge. Answer questions comprehensively " "drawing on your extensive training. The reference material below is just a starting point — " "feel free to expand with additional background knowledge beyond what's provided. " "Make the answer as rich and informative as possible."), ("human", "Reference material:\n{context}\n\nQuestion: {question}\n\nProvide a comprehensive answer:"), ])

**Root Cause**: The prompt explicitly authorizes external knowledge injection. RAGAS `faithfulness` flags any claim not traceable to `{context}`, causing a sharp metric drop despite intact retrieval.

### Failure Mode 3: Off-Topic Answers

p3_pipeline = RAGPipeline( chunk_size=512, chunk_overlap=50, top_k=4, prompt_type="offtopic", # Forced academic survey format )

PROMPT_OFFTOPIC = ChatPromptTemplate.from_messages([ ("system", "You are a senior technical researcher writing academic surveys. " "For every question, structure your response exactly as follows:\n" "1. Technical Background and Historical Evolution\n" "2. Major Technical Schools and Comparative Analysis\n" "3. Current Challenges and Future Trends\n" "Each section must be at least 200 words. Use academic language."), ("human", "Reference material:\n{context}\n\nQuestion: {question}\n\nWrite the survey:"), ])

**Root Cause**: Rigid structural mandates override direct question answering. `answer_relevancy` measures directness; forced academic formatting decouples output from user intent.

### Remediation Configurations

Before: chunks too small, top_k too low

RAGPipeline(chunk_size=64, chunk_overlap=0, top_k=1)

After: reasonable chunk size + sufficient top_k

RAGPipeline(chunk_size=512, chunk_overlap=50, top_k=4)

**Reference Values:**
| Use Case | chunk_size | overlap | top_k |
|----------|------------|---------|-------|
| Short Q&A (technical FAQ) | 256–512 | 20–50 | 3–5 |
| Long document comprehension | 512–1024 | 50–100 | 4–6 |
| Code repository search | Per function/class | 0 | 3–5 |

**Prompt Hardening Principle**: Explicitly constrain the model to the reference material and define fallback behavior for insufficient context.

PROMPT_STRICT = Cha

## Pitfall Guide
1. **Intuition-Driven Tuning**: Modifying prompts or embeddings without baseline metrics creates confounded variables. Always run RAGAS evaluation before and after changes to isolate impact.
2. **Fragmented Chunking**: Setting `chunk_size` below semantic boundaries (e.g., <128 tokens) shatters context. `context_recall` will drop regardless of embedding quality. Align chunk size with concept density.
3. **Prompt Leakage**: Phrases like "feel free to expand" or "use your general knowledge" break grounding. RAG requires explicit constraints: "Answer strictly based on provided context."
4. **Over-Structured Prompts**: Forcing rigid templates (academic, bullet-heavy, word-count mandates) decouples output from user intent. Prioritize directness over formatting compliance.
5. **Ignoring Context Precision**: Low `context_recall` often coincides with low `context_precision`. Retrieving tiny, topically relevant but semantically empty fragments wastes LLM context windows and degrades generation quality.
6. **Missing Fallback Instructions**: Failing to instruct the model to explicitly state "insufficient information" when context lacks answers forces hallucination or irrelevant padding. Always include a negative constraint clause.

## Deliverables
- **📘 RAG Diagnostic Blueprint**: Step-by-step decision tree mapping RAGAS metric thresholds to pipeline components (Retrieval vs. Generation). Includes metric sensitivity analysis and remediation pathways.
- **✅ Pre-Deployment RAG Validation Checklist**: 12-point verification protocol covering chunking strategy, prompt grounding constraints, `top_k` calibration, and RAGAS baseline thresholds before production rollout.
- **⚙️ Configuration Templates**: Production-ready JSON/YAML templates for `RAGPipeline` parameters (chunk_size, overlap, top_k) by use case, plus hardened prompt templates with explicit grounding and fallback clauses.