AI caching and response optimization

ery_vector AS score]`; const results = await this.redis.ft.search(this.indexName, query, { PARAMS: { query_vector: Float32Array.from(embedding) }, RETURN: ['$.response', '$.promptHash', '$.metadata'], LIMIT: { from: 0, size: 1 }, } as any);

if (results.total > 0) {
  const doc = results.documents[0];
  const score = (doc as any).score;
  if (score >= this.threshold) {
    const cached = JSON.parse(doc.json as string);
    return cached['$.response'];
  }
}

// Cache miss: compute and store
const response = await computeFn(prompt);
await this.store(normalized, promptHash, embedding, response, options);
return response;

}

private async store( prompt: string, promptHash: string, embedding: number[], response: string, options: { ttl?: number; contextVersion?: string } ) { const key = ${this.indexName}:${promptHash}; const ttl = options.ttl ?? 3600; const entry = { promptHash, embedding, response, metadata: { contextVersion: options.contextVersion ?? 'v1' }, createdAt: Date.now(), ttl, }; await this.redis.json.set(key, '$', entry); await this.redis.expire(key, ttl); }

private normalizePrompt(prompt: string): string { return prompt .toLowerCase() .replace(/\b\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}\b/g, '[DATE]') .replace(/\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b/g, '[UUID]') .replace(/\s+/g, ' ') .trim(); }

private hashPrompt(prompt: string, contextVersion?: string): string { const crypto = require('crypto'); const base = ${prompt}|${contextVersion ?? 'default'}; return crypto.createHash('sha256').update(base).digest('hex').slice(0, 16); } }

### Architecture Decisions & Rationale

- **Redis with RediSearch over pgvector**: Redis delivers sub-5ms vector lookups with built-in TTL, clustering, and JSON storage. pgvector adds query planning overhead and lacks native expiration. For AI caching, speed and lifecycle management outweigh relational consistency.
- **Cosine Similarity over Dot Product**: Normalized embeddings make cosine distance invariant to magnitude. AI prompts vary in length and token density; cosine ensures semantic equivalence isn't penalized by vector scale.
- **Threshold Calibration at 0.92**: Below 0.90, semantic drift introduces context mismatches. Above 0.95, hit rates collapse. 0.92 balances precision and recall across instruction-tuned models. Dynamic thresholding based on request volume should be implemented in production monitoring.
- **Prompt Versioning via Context Hash**: System prompts, model versions, and tool definitions change response behavior. Including a `contextVersion` in the hash ensures cache invalidation when inference parameters shift, preventing stale-context poisoning.
- **Normalization Strategy**: Stripping timestamps, UUIDs, and session identifiers preserves intent while eliminating byte-level noise. This is critical for chat and ticketing workflows where identical requests arrive with rotating metadata.

## Pitfall Guide

1. **Static Threshold Rigidity**: Hardcoding `0.92` without monitoring hit rates and latency tradeoffs leads to either cache starvation or response drift. Implement adaptive thresholds that adjust based on p95 latency targets and cost budgets.
2. **Ignoring System Prompt Variations**: Caching only user prompts while system prompts change between calls causes context mismatch. Always include system prompt hash, model identifier, and tool definitions in the cache key.
3. **Caching Streaming Responses as Blobs**: Streaming APIs emit chunks with different byte signatures. Caching the full stream as a single string breaks delta validation. Store streaming responses as concatenated final text, or cache at the chunk level with explicit boundaries.
4. **Embedding Cost Blindness**: Generating embeddings adds ~$0.02 per 1k tokens. If your hit rate drops below 25%, embedding overhead may exceed inference savings. Profile embedding latency/cost before scaling semantic caching to high-throughput endpoints.
5. **Context Drift & Knowledge Cutoff Mismatch**: Cached responses reflect the model's state at cache time. If the LLM is upgraded or knowledge cutoff shifts, stale answers persist. Implement context versioning and periodic cache warmups with fresh inference.
6. **Cache Poisoning via Adversarial Prompts**: Malicious inputs can inject false vectors into the cache. Validate prompt structure, enforce rate limits on cache writes, and monitor cosine similarity distributions for anomalous clustering.
7. **Over-Indexing High-Cardinality Variables**: Caching prompts with user IDs, request IDs, or session tokens defeats semantic matching. Normalize or strip these fields before embedding. Keep cache keys intent-focused, not identity-focused.

**Best Practices from Production**:
- Run A/B cache hit rate tracking against latency and cost dashboards.
- Use embedding quantization (FP16 or INT8) to reduce memory footprint without significant accuracy loss.
- Implement fallback routing: if vector search fails or latency exceeds 50ms, bypass cache and call LLM directly.
- Cache invalidation should be event-driven, not TTL-only. Invalidate on system prompt changes, model upgrades, or knowledge base updates.

## Production Bundle

### Action Checklist
- [ ] Normalize prompts: strip timestamps, UUIDs, session tokens, and high-cardinality metadata before embedding
- [ ] Implement context versioning: hash system prompts, model IDs, and tool definitions into cache keys
- [ ] Calibrate similarity threshold: start at 0.92, monitor hit rate vs p95 latency, adjust dynamically
- [ ] Add embedding cost monitoring: track $/1k requests with and without cache to validate ROI
- [ ] Implement TTL + event-driven invalidation: expire stale entries, purge on model/system prompt updates
- [ ] Secure cache writes: rate-limit embedding generation, validate prompt structure, monitor similarity anomalies
- [ ] Instrument hit/miss routing: log cache decisions, latency deltas, and cost savings per endpoint
- [ ] Test streaming compatibility: ensure cached responses align with delta validation and chunk boundaries

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| High-volume support bot with repetitive intents | Semantic Cache (θ=0.92) | 40–50% hit rate absorbs redundant queries | Reduces inference spend by 60–70% |
| Real-time code assistant with unique context | Exact-Match + Short TTL | Low semantic redundancy, high context sensitivity | Minimal cost change, prevents duplicate calls |
| Document summarization with knowledge cutoff updates | Semantic Cache + Context Versioning | Prevents stale answers while capturing intent repeats | Moderate embedding cost, high accuracy preservation |
| Low-traffic internal tool (<100 req/min) | No Cache or Exact-Match | Overhead of vector search outweighs savings | Neutral to slightly positive ROI |
| Multi-model routing with frequent upgrades | Semantic Cache + Event Invalidation | Ensures cache aligns with current model behavior | Prevents regression costs from stale responses |

### Configuration Template

```typescript
// ai-cache.config.ts
export const aiCacheConfig = {
  redis: {
    url: process.env.REDIS_URL || 'redis://localhost:6379',
    maxRetries: 3,
    retryDelay: 100,
  },
  semantic: {
    threshold: 0.92,
    embeddingModel: 'text-embedding-3-small',
    dimensions: 1536,
    indexName: 'ai_semantic_cache_v2',
  },
  lifecycle: {
    defaultTTL: 3600, // seconds
    contextVersion: process.env.AI_CONTEXT_VERSION || 'v1',
    invalidationEvents: ['MODEL_UPGRADE', 'SYSTEM_PROMPT_CHANGE', 'KB_UPDATE'],
  },
  routing: {
    fallbackOnVectorFailure: true,
    maxVectorLatencyMs: 50,
    bypassThreshold: 0.85, // skip cache if similarity < this
  },
  monitoring: {
    enabled: true,
    metricsPrefix: 'ai.cache',
    alertOnHitRateBelow: 0.25,
    alertOnP95LatencyAboveMs: 300,
  },
};

Quick Start Guide

1. Install dependencies: npm install redis @anthropic-ai/sdk openai (or your preferred embedding/inference provider)
2. Initialize Redis with RediSearch: Deploy Redis 7+ with the RediSearch module enabled. Verify FT.CREATE commands succeed.
3. Instantiate the cache: Import SemanticAICache, pass your Redis URL and embedding client, call await cache.connect()
4. Wrap LLM calls: Replace direct inference calls with await cache.getOrCompute(prompt, async (p) => llm.generate(p), { ttl: 3600, contextVersion: 'v1' })
5. Monitor & calibrate: Track hit rate, latency delta, and cost per 1k requests. Adjust threshold and defaultTTL based on workload patterns. Deploy context versioning before model upgrades.