Day 11: Conversational RAG — How to Chat with Your Documents 💬

lone, search-optimized queries.

• The sweet spot balances retrieval precision and system latency by decoupling context comprehension from document fetching.
• Explicit state management (chat_history mutation) is the critical differentiator between broken and production-ready conversational RAG.

Core Solution

The architecture introduces a dedicated contextualization layer that intercepts user input, cross-references it with conversation state, and outputs a semantically complete query for the retriever. This is implemented using LangChain's create_history_aware_retriever and chained with a standard QA pipeline.

Step 1: Contextualizing the Question

A sub-chain evaluates the latest input against prior turns to produce a retriever-friendly query. The system prompt explicitly forbids answering, ensuring the LLM acts purely as a query transformer.

from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

# The prompt that tells the AI to re-write the question if history exists
contextualize_q_system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question which can be understood "
    "without the chat history. Do NOT answer the question."
)

contextualize_q_prompt = ChatPromptTemplate.from_messages([
    ("system", contextualize_q_system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}"),
])

# Wrap your existing retriever (from Day 9)
history_aware_retriever = create_history_aware_retriever(
    llm, retriever, contextualize_q_prompt
)

Step 2: The Full Conversational Chain

The rewritten query flows into the retrieval step, and the resulting context is passed to a QA chain that maintains the conversation thread for final answer generation.

from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain

# Standard Q&A prompt
qa_system_prompt = (
    "You are an assistant for question-answering tasks. "
    "Use the following pieces of retrieved context to answer the question."
    "\n\n"
    "{context}"
)

qa_prompt = ChatPromptTemplate.from_messages([
    ("system", qa_system_prompt),
    MessagesPlaceholder("chat_history"),
    ("human", "{input}"),
])

question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)

# The final chain!
rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

Testing it Out

State management must be explicitly handled at the application layer. The chat_history list is mutated after each turn to preserve the bidirectional conversation flow.

from langchain_core.messages import HumanMessage, AIMessage

chat_history = []

# First Interaction
question = "What is LangSmith?"
result = rag_chain.invoke({"input": question, "chat_history": chat_history})
print(result["answer"])

# Update History
chat_history.extend([
    HumanMessage(content=question),
    AIMessage(content=result["answer"]),
])

# Follow-up (The AI now knows 'it' refers to LangSmith!)
second_question = "How do I get started with it?"
result = rag_chain.invoke({"input": second_question, "chat_history": chat_history})
print(result["answer"])

Pitfall Guide

1. Neglecting Explicit History Mutation: The chain does not auto-manage state. Failing to extend() the chat_history list with HumanMessage and AIMessage after each invocation causes the retriever to receive an empty context, reverting the system to isolated retrieval.
2. Unbounded Context Window Growth: Continuously appending messages without truncation or summarization will eventually exceed the LLM's token limit, triggering ContextWindowExceeded errors or severe performance degradation. Implement sliding windows, token-based truncation, or periodic history summarization.
3. Prompt Leakage in Rewriting Step: Omitting "Do NOT answer the question" in contextualize_q_system_prompt allows the LLM to generate a direct response instead of a standalone query. This breaks the retrieval pipeline and wastes compute on premature generation.
4. Semantic Drift from Aggressive Rewriting: Overly creative LLM rewriting can alter user intent or inject hallucinated constraints. Always log the original input alongside the rewritten query for auditability, and consider temperature constraints (temperature=0.1) on the contextualization LLM.
5. Ignoring Empty History Edge Cases: On the first turn, chat_history is an empty list. The retriever must gracefully handle this without throwing type errors or injecting placeholder tokens that confuse the vector database. LangChain's MessagesPlaceholder handles this natively, but custom implementations must validate list state.
6. Latency Compounding in High-Throughput Pipelines: Adding a rewrite step introduces an extra LLM call (~100–200ms). For latency-sensitive applications, cache rewritten queries for identical follow-ups, or offload the contextualization step to a smaller, faster model (e.g., distilled LLM) while reserving larger models for final QA generation.

Deliverables

• Blueprint: Conversational RAG Architecture Flowchart & State Management Diagram (PDF/Draw.io)
• Checklist: Pre-Deployment Validation for History-Aware Retrieval (Context Truncation, Prompt Guardrails, Latency Benchmarks, Error Handling)
• Configuration Templates: LangChain Prompt Templates (contextualize_q, qa_system), Retriever Configuration Snippets, and History Sliding Window Implementation Code