LLM multi-turn conversations

for (const msg of [...recentMessages, newMessage]) {
  const tokens = await this.countTokens(msg.content);
  if (totalTokens + tokens > state.tokenBudget * 0.85) {
    break; // Reserve buffer for assistant response
  }
  contextWindow.push(msg);
  totalTokens += tokens;
}

// Inject compressed memory if relevant
const memoryContext = this.formatMemory(state.memory);
if (memoryContext) {
  const memTokens = await this.countTokens(memoryContext);
  if (totalTokens + memTokens < state.tokenBudget * 0.85) {
    contextWindow.splice(1, 0, { role: 'system', content: memoryContext, timestamp: Date.now(), turnId: 'mem' });
  }
}

return contextWindow;

}

async updateState(state: ConversationState, assistantResponse: string, turnId: string): Promise<ConversationState> { const newState = { ...state, messages: [...state.messages], memory: { ...state.memory }, version: state.version + 1 };

newState.messages.push({ role: 'assistant', content: assistantResponse, timestamp: Date.now(), turnId });

// Trigger compression if budget exceeded
const totalTokens = await this.estimateStateTokens(newState);
if (totalTokens > state.tokenBudget * this.COMPRESSION_THRESHOLD) {
  newState.memory = await this.compressContext(newState.messages);
  newState.messages = newState.messages.slice(-6); // Keep recent turns
}

return newState;

}

private extractSystemPrompt(state: ConversationState): string { const constraints = Object.entries(state.memory) .filter(([k]) => k.startsWith('constraint:')) .map(([, v]) => v) .join('\n'); return You are a persistent assistant. Maintain all active constraints:\n${constraints || 'None specified.'}; }

private formatMemory(memory: Record<string, string>): string | null { const entries = Object.entries(memory) .filter(([k]) => !k.startsWith('constraint:')) .map(([k, v]) => ${k}: ${v}) .join('; '); return entries ? Session Memory: ${entries} : null; }

private async compressContext(messages: Message[]): Promise<Record<string, string>> { const memory: Record<string, string> = {}; // In production, replace with LLM-based summarization or embedding similarity routing const recent = messages.slice(-4); memory['summary'] = recent.map(m => [${m.role}] ${m.content.slice(0, 100)}).join(' | '); memory['active_goals'] = 'Resolve user query while preserving prior constraints.'; return memory; }

private async estimateStateTokens(state: ConversationState): Promise<number> { let total = 0; for (const msg of state.messages) total += await this.countTokens(msg.content); for (const v of Object.values(state.memory)) total += await this.countTokens(v); return total; } }

### Architecture Decisions & Rationale

- **Decoupled State Store:** Conversation state lives outside the LLM API call. This enables rollback, auditability, and multi-model routing without re-architecting the chat flow.
- **Token Budgeting Over Hard Limits:** Models support large contexts, but performance degrades near the ceiling. Budgeting at 85% reserves headroom for assistant generation and prevents silent truncation.
- **Semantic Compression Over Naive Truncation:** Archiving older turns into summaries preserves constraints and user preferences while reducing token load. Embedding similarity ensures only relevant memory is injected.
- **Streaming State Reconciliation:** Pending states prevent UI/backend desync. If a stream fails, the system reverts to the last committed turn, avoiding partial context corruption.
- **Versioned State:** Incremental versioning enables conflict resolution in distributed deployments and supports optimistic updates with deterministic rollbacks.

## Pitfall Guide

### 1. Blind History Accumulation
Appending every turn without pruning causes context dilution. Models attend less to earlier tokens, and instruction drift becomes inevitable. Production systems must enforce sliding windows and active compression.

### 2. Ignoring Token Budget Allocation
Treating the context window as a free pool leads to API errors or silent truncation. Reserve 15–20% for assistant generation, system prompts, and memory injection. Calculate tokens per turn, not per session.

### 3. State Leakage Across Sessions
Reusing memory objects or failing to isolate session IDs causes cross-contamination. Users receive responses tailored to other conversations. Implement strict session boundaries and cryptographic session tokens.

### 4. Over-Compression Losing Constraints
Aggressive summarization discards negative constraints ("do not use Python", "avoid financial advice"). Always separate constraints from factual summaries. Inject constraints into the system prompt regardless of compression.

### 5. Streaming State Desync
Displaying partial tokens while the backend tracks full turns creates state mismatches. If the stream drops, the UI shows incomplete context while the backend expects a full turn. Commit assistant state only after stream completion or explicit user acknowledgment.

### 6. Assuming Positional Neutrality
LLMs are highly sensitive to token position. Critical instructions placed in compressed memory or buried in long history lose weight. Place active constraints at the top of the context window. Repeat non-negotiable rules in the system prompt.

### Production Best Practices
- Token-count every message before injection
- Maintain separate tracks for raw logs, active context, and compressed memory
- Use idempotent turn IDs for retry safety
- Log compression ratios and context retention scores for tuning
- Implement circuit breakers when token spend exceeds thresholds per session

## Production Bundle

### Action Checklist
- [ ] Define conversation state schema with raw messages, structured memory, and token budget
- [ ] Implement tokenizer-aligned token counting before every context assembly
- [ ] Enforce sliding window limits with semantic compression for archived turns
- [ ] Separate constraints from factual summaries to prevent instruction loss
- [ ] Add streaming reconciliation with pending state and rollback on failure
- [ ] Instrument context retention, token efficiency, and compression ratio metrics
- [ ] Isolate session boundaries with cryptographic IDs and versioned state
- [ ] Reserve 15–20% token budget for assistant generation and system injection

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Customer Support Chat | Sliding Window + Constraint Injection | High constraint sensitivity, short resolution cycles | -32% vs naive |
| Code Generation Assistant | Structured Memory + Semantic Compression | Requires persistent context, tool state, and file references | -48% vs naive |
| Creative Writing / Roleplay | Naive Appending (Limited Turns) | Narrative flow benefits from full history, low constraint density | +15% vs baseline |
| Enterprise Knowledge Retrieval | Structured Memory + RAG Overlay | Factual accuracy requires external grounding, not raw history | -41% vs naive |
| Multi-Agent Orchestration | Event-Sourced State + Versioned Context | Deterministic replay, audit trails, and agent handoff require strict state | -28% vs naive |

### Configuration Template

```typescript
// conversation.config.ts
export const ConversationConfig = {
  model: 'gpt-4o',
  tokenBudget: 12000,
  maxRecentTurns: 8,
  compressionThreshold: 0.75,
  streamingBufferSize: 128,
  stateBackend: {
    type: 'redis',
    ttl: 3600, // 1 hour session expiry
    keyPrefix: 'conv:',
    serialization: 'json'
  },
  observability: {
    trackTokenSpend: true,
    trackCompressionRatio: true,
    alertOnBudgetExceed: true,
    logContextRetention: true
  },
  fallback: {
    onStateDesync: 'rollback_last_committed',
    onTokenOverflow: 'compress_and_retry',
    onStreamFailure: 'discard_pending_and_notify'
  }
};

Quick Start Guide

1. Install dependencies: npm install js-tiktoken ioredis uuid
2. Initialize the manager: const manager = new MultiTurnManager('gpt-4o');
3. Create initial state: const state = { sessionId: uuidv4(), messages: [], memory: {}, tokenBudget: 12000, usedTokens: 0, version: 0 };
4. Process first turn: const context = await manager.createContext(state, { role: 'user', content: 'Hello', timestamp: Date.now(), turnId: uuidv4() });
5. Stream response and update: const response = await callLLM(context); const newState = await manager.updateState(state, response, turnId);

Run this flow in a loop. Monitor token spend and compression ratios. Adjust tokenBudget and maxRecentTurns based on workload constraints. The system will maintain context fidelity while controlling latency and cost.