RAG architecture patterns

[]> { // In production, replace with a hosted cross-encoder or local Cohere/BGE-reranker const scored = docs.map(doc => ({ doc, score: this.heuristicRelevance(query, doc.pageContent) }));

const filtered = scored
  .filter(item => item.score >= this.config.reRankThreshold)
  .sort((a, b) => b.score - a.score)
  .map(item => item.doc);

return filtered.slice(0, this.config.topK);

}

private heuristicRelevance(query: string, content: string): number { const qWords = new Set(query.toLowerCase().split(/\W+/)); const cWords = content.toLowerCase().split(/\W+/); let matches = 0; for (const w of cWords) if (qWords.has(w)) matches++; return matches / Math.max(qWords.size, 1); }

private compressContext(docs: Document[]): string { // Production: replace with LLM-based summarization or extractive compression return docs.map(d => d.pageContent).join("\n\n").slice(0, this.config.maxContextTokens * 4); }

private buildPrompt(context: string, query: string): string { return `You are a precise technical assistant. Answer using ONLY the provided context. Context: ${context}

Question: ${query} Answer:`; }

async run(query: string): Promise<PipelineResult> { const rawRetrieved = await this.hybridSearch(query); const reRanked = await this.reRank(rawRetrieved, query); const compressed = this.compressContext(reRanked); const prompt = this.buildPrompt(compressed, query);

const chain = RunnableSequence.from([
  new RunnablePassthrough(),
  this.llm,
  this.outputParser
]);

const generation = await chain.invoke(prompt);

return {
  query,
  retrieved: rawRetrieved,
  reRanked,
  compressedContext: compressed,
  generation,
  metadata: {
    retrievalCount: rawRetrieved.length,
    postReRankCount: reRanked.length,
    contextTokens: Math.ceil(compressed.length / 4),
    timestamp: new Date().toISOString()
  }
};

} }

### Architectural Rationale
- **Separation of Retrieval and Ranking**: Hybrid search casts a wide net; re-ranking narrows it. This decoupling allows independent tuning of recall vs precision.
- **Explicit Context Boundaries**: `maxContextTokens` prevents unbounded expansion. Production systems must enforce hard limits before LLM ingestion.
- **Observable Envelopes**: Every stage returns metadata. This enables downstream evaluation pipelines (RAGAS, Arize Phoenix) to trace degradation to specific stages.
- **Replaceable Components**: The re-ranker and compressor are abstracted. Swapping heuristic scoring for a cross-encoder or LLM-based compressor requires zero pipeline restructuring.

## Pitfall Guide

1. **Fixed-Size Chunking Without Overlap or Semantic Awareness**
   Splitting documents at exact byte/word boundaries fractures tables, code blocks, and logical arguments. Production impact: retrieval precision drops 15–20% because chunks lose contextual anchors.
   *Mitigation*: Use recursive character splitting with 10–15% overlap, or semantic chunking via sentence boundary detection + embedding similarity thresholds.

2. **Vector-Only Search for Technical/Domain Content**
   Embeddings struggle with exact identifiers, version numbers, API endpoints, and numerical constraints. Relying solely on cosine similarity guarantees lexical misses.
   *Mitigation*: Always pair dense retrieval with BM25 or SPLADE. Weight lexical matches higher for query types containing alphanumeric patterns.

3. **Skipping Re-Ranking**
   Top-K vector results contain high-recall but low-precision candidates. Feeding noise into the LLM wastes tokens and degrades attention allocation.
   *Mitigation*: Insert a cross-encoder re-ranker (BGE-reranker, Cohere, or ColBERT) between retrieval and generation. Expect 200–400ms overhead for 30–40% precision gains.

4. **Unbounded Context Windows**
   Developers assume "more context = better answers". LLMs exhibit lost-in-the-middle degradation and attention dilution beyond 8k–12k tokens.
   *Mitigation*: Enforce strict token budgets. Compress, summarize, or extract before generation. Route long-context queries to specialized pipelines.

5. **Static Embeddings Without Incremental Updates**
   Knowledge drift is inevitable. Re-embedding entire corpora daily is cost-prohibitive and causes index fragmentation.
   *Mitigation*: Implement append-only ingestion with document versioning. Use hybrid incremental updates: only re-embed changed sections. Maintain tombstone markers for deleted content.

6. **Missing Evaluation Loop**
   Accuracy is assumed, not measured. Without RAG-specific metrics, teams cannot distinguish retrieval failure from generation failure.
   *Mitigation*: Deploy automated evaluation using RAGAS or proprietary rubrics. Track context precision, answer relevance, and faithfulness per pipeline version. Gate deployments on metric thresholds.

7. **Ignoring Query Type Distribution**
   Not all queries require the same pipeline. Factual lookups, multi-hop reasoning, and comparative analysis demand different retrieval strategies.
   *Mitigation*: Implement query classification routing. Direct simple lookups to lightweight pipelines, complex reasoning to modular/graph patterns. Log distribution shifts to adjust routing weights.

## Production Bundle

### Action Checklist
- [ ] Define query type taxonomy: Classify expected queries into factual, comparative, procedural, and multi-hop categories to drive routing decisions.
- [ ] Implement hybrid retrieval baseline: Deploy BM25 + dense vector search with weighted scoring before adding re-rankers or graph layers.
- [ ] Enforce context token budgets: Set hard limits on pre-generation context and implement compression or extraction fallbacks.
- [ ] Insert cross-encoder re-ranking: Replace heuristic scoring with a production re-ranker and validate precision gains on holdout sets.
- [ ] Instrument pipeline observability: Log retrieval counts, re-rank scores, context token usage, and generation latency per request.
- [ ] Deploy automated evaluation: Integrate RAGAS or custom rubrics to track context precision, faithfulness, and answer relevance on every deployment.
- [ ] Establish incremental embedding workflow: Version documents, track change deltas, and re-embed only modified segments to control cost and drift.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Internal FAQ / SaaS documentation | Advanced RAG (Hybrid + Re-rank) | High lexical precision needed; re-ranker filters noise efficiently | +15% vs naive, -40% support tickets |
| Multi-step troubleshooting / policy reasoning | Modular RAG (Self-Correction + Multi-hop) | Requires iterative retrieval and verification steps | +35% compute, -60% hallucination |
| Cross-document synthesis / regulatory compliance | Graph RAG + Vector | Entity relationships and citation tracing prevent fabricated links | +55% infrastructure, +90% auditability |
| Low-latency customer chat (<500ms) | Lightweight Hybrid + Prompt Compression | Re-rankers exceed latency budget; compression preserves signal | +8% cost, meets SLA thresholds |

### Configuration Template

```typescript
// rag.config.ts
export interface RAGPipelineConfig {
  mode: 'advanced' | 'modular' | 'graph';
  retrieval: {
    topK: number;
    hybridWeights: { vector: number; lexical: number };
    reRank: {
      enabled: boolean;
      model: string;
      threshold: number;
      maxCandidates: number;
    };
  };
  context: {
    maxTokens: number;
    compression: 'extract' | 'summarize' | 'none';
    chunkOverlapPercent: number;
  };
  generation: {
    model: string;
    temperature: number;
    maxTokens: number;
    fallbackModel?: string;
  };
  observability: {
    logRawRetrieval: boolean;
    metricEndpoint: string;
    evaluationThresholds: {
      contextPrecision: number;
      faithfulness: number;
    };
  };
}

export const defaultConfig: RAGPipelineConfig = {
  mode: 'advanced',
  retrieval: {
    topK: 8,
    hybridWeights: { vector: 0.6, lexical: 0.4 },
    reRank: {
      enabled: true,
      model: 'BAAI/bge-reranker-v2-m3',
      threshold: 0.72,
      maxCandidates: 32
    }
  },
  context: {
    maxTokens: 4096,
    compression: 'extract',
    chunkOverlapPercent: 12
  },
  generation: {
    model: 'gpt-4o-mini',
    temperature: 0.1,
    maxTokens: 1024,
    fallbackModel: 'claude-3-haiku'
  },
  observability: {
    logRawRetrieval: false,
    metricEndpoint: '/api/v1/metrics/rag',
    evaluationThresholds: {
      contextPrecision: 0.75,
      faithfulness: 0.80
    }
  }
};

Quick Start Guide

1. Initialize vector store and ingest sample documents: Run the embedding pipeline on a 500-document subset. Configure recursive chunking with 12% overlap and load into Chroma/Pinecone/Weaviate.
2. Deploy hybrid retrieval layer: Instantiate BM25 and vector retrievers. Apply the defaultConfig hybrid weights. Validate recall on 50 held-out queries.
3. Attach re-ranker and context compressor: Enable the cross-encoder re-ranker. Set maxTokens: 4096 and compression: extract. Run latency benchmark; ensure P95 stays under 800ms.
4. Connect LLM and observability: Wire the generation stage to your model provider. Enable metric logging to /api/v1/metrics/rag. Execute evaluation suite; gate deployment if contextPrecision < 0.75.