AI caching and response optimization

): Promise<string> { const normalized = compressPrompt(prompt); const embedding = await generateEmbedding(normalized); const cacheKey = ai:cache:${this.hash(normalized)};

// Request coalescing to prevent stampede
const lockKey = `ai:lock:${this.hash(normalized)}`;
const lockAcquired = await this.redis.set(lockKey, '1', { NX: true, EX: 5 });

if (!lockAcquired) {
  // Wait for concurrent request to resolve
  return this.waitForCache(cacheKey, 3000);
}

try {
  const cached = await this.findSemanticMatch(embedding);
  if (cached) {
    await this.redis.hIncrBy(cacheKey, 'hitCount', 1);
    return cached.response;
  }

  // Fallback to LLM generation
  const response = await this.generateResponse(normalized);
  const tokens = tokenize(response).length;
  
  if (tokens > this.maxTokens) {
    return response.slice(0, this.maxTokens * 4); // rough char approximation
  }

  const ttl = this.calculateDynamicTTL(tokens);
  await this.redis.hSet(cacheKey, {
    prompt: normalized,
    response,
    embedding: JSON.stringify(embedding),
    tokens: String(tokens),
    createdAt: String(Date.now()),
    ttl: String(ttl),
    hitCount: '0'
  });
  await this.redis.expire(cacheKey, ttl);

  return response;
} finally {
  await this.redis.del(lockKey);
}

}

private async findSemanticMatch(queryEmbedding: number[]): Promise<CacheEntry | null> { const keys = await this.redis.keys('ai:cache:*'); let bestMatch: CacheEntry | null = null; let bestScore = -1;

for (const key of keys) {
  const raw = await this.redis.hGetAll(key);
  if (!raw.embedding) continue;
  
  const storedEmbedding = JSON.parse(raw.embedding);
  const score = cosineSimilarity(queryEmbedding, storedEmbedding);
  
  if (score >= this.similarityThreshold && score > bestScore) {
    bestScore = score;
    bestMatch = {
      prompt: raw.prompt,
      response: raw.response,
      embedding: storedEmbedding,
      tokens: Number(raw.tokens),
      createdAt: Number(raw.createdAt),
      ttl: Number(raw.ttl),
      hitCount: Number(raw.hitCount)
    };
  }
}

return bestMatch;

}

private calculateDynamicTTL(tokens: number): number { // Shorter TTL for high-token responses to reduce stale cache risk const decay = Math.max(0.5, 1 - (tokens / this.maxTokens) * 0.3); return Math.floor(this.baseTTL * decay); }

private async generateResponse(prompt: string): Promise<string> { // Replace with your LLM provider SDK // Supports streaming internally, but returns aggregated string for cache storage throw new Error('Implement LLM generation'); }

private async waitForCache(key: string, timeout: number): Promise<string> { const start = Date.now(); while (Date.now() - start < timeout) { const exists = await this.redis.exists(key); if (exists) { const raw = await this.redis.hGetAll(key); return raw.response; } await new Promise(r => setTimeout(r, 100)); } throw new Error('Cache wait timeout'); }

private hash(input: string): string { return Buffer.from(input).toString('base64').replace(/[^a-zA-Z0-9]/g, '').slice(0, 16); } }

### Architecture Decisions & Rationale
- **Semantic matching over exact keys**: LLM prompts vary syntactically but converge semantically. Cosine similarity on embeddings captures intent equivalence without manual regex or prompt normalization.
- **Dynamic TTL tied to token count**: High-token responses consume more cache memory and age faster. Reducing TTL proportionally limits stale data exposure.
- **Request coalescing via distributed locks**: Prevents redundant LLM calls during cache misses. Promise deduplication ensures only one inference triggers per semantic cluster.
- **Cache-aside with streaming passthrough**: The cache stores aggregated responses, but production gateways should stream directly to clients while buffering for cache writes. This decouples latency from cache write latency.

## Pitfall Guide

1. **Relying exclusively on exact-match caching**
   LLM prompts are naturally paraphrased. Exact-match caches achieve <25% hit rates in production. Semantic vectors or fuzzy hashing must replace string equality.

2. **Ignoring context drift and temporal data**
   Cached responses become stale when underlying facts change (pricing, policies, system states). Implement versioned cache keys or attach data freshness metadata to cache entries.

3. **Caching system-state-dependent prompts**
   Prompts containing user IDs, session tokens, or real-time metrics should never be cached. Filter dynamic segments before embedding generation.

4. **Cache stampede during peak loads**
   Without request coalescing, 100 concurrent identical requests trigger 100 LLM calls. Distributed locks or in-memory promise deduplication are mandatory.

5. **Neglecting streaming optimization**
   Caching aggregated responses breaks streaming UX. Implement a dual-path architecture: stream directly to the client while writing to cache asynchronously. Subsequent requests receive the cached buffer.

6. **Static TTL without usage analytics**
   Fixed expiration ignores traffic patterns. Cache entries with high hit counts should receive TTL extensions; low-traffic entries should expire faster to reclaim memory.

7. **Arbitrary similarity thresholds**
   A threshold of 0.75 may match unrelated prompts; 0.95 may miss valid duplicates. Calibrate thresholds using a validation set of known duplicate prompts and measure precision/recall tradeoffs.

**Best Practices from Production**
- Run a cache warming job during off-peak hours to pre-populate high-frequency semantic clusters.
- Monitor cache hit rate, P95 latency, and token expenditure per 1k requests. Alert when hit rate drops below 60%.
- Use approximate nearest neighbor (ANN) indexes like HNSW or FAISS for vector search when cache size exceeds 10k entries.
- Implement cache invalidation webhooks for data source changes. Trigger semantic re-validation instead of blanket flushes.

## Production Bundle

### Action Checklist
- [ ] Deploy semantic embedding pipeline: Integrate a lightweight embedding model to convert prompts into dense vectors before cache lookup.
- [ ] Configure similarity threshold: Set cosine similarity to 0.82–0.87 and validate against a labeled duplicate prompt dataset.
- [ ] Implement request coalescing: Add distributed locks or promise deduplication to prevent cache stampede during misses.
- [ ] Apply dynamic TTL: Tie expiration to token count and hit frequency to reduce stale response exposure.
- [ ] Enable streaming passthrough: Stream LLM output directly to clients while asynchronously writing aggregated responses to cache.
- [ ] Monitor cache telemetry: Track hit rate, P95 latency, token spend, and invalidation frequency. Alert on degradation.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| High-volume FAQ/chatbot | Semantic Vector Cache + ANN Index | Prompts are highly repetitive; vector search captures intent equivalence at scale. | Reduces API spend by 55–70% |
| Real-time data queries | Token-Aware Hybrid + Versioned Keys | Data freshness matters; semantic caching with TTL decay prevents stale outputs. | Moderate cost increase for validation, but avoids incorrect responses |
| Low-traffic internal tools | Exact-Match + Short TTL | Overhead of embedding generation outweighs benefits; simple caching suffices. | Minimal infrastructure cost, ~15–20% savings |
| Multi-turn conversational UI | Session-Scoped Cache + Prompt Compression | Context window grows with turns; compress history and cache turn-level responses. | Reduces context token waste by 40–50% |

### Configuration Template

```json
{
  "aiCache": {
    "redis": {
      "url": "redis://localhost:6379",
      "keyPrefix": "ai:cache:",
      "maxEntries": 50000,
      "evictionPolicy": "volatile-lru"
    },
    "semantic": {
      "embeddingModel": "text-embedding-3-small",
      "similarityThreshold": 0.85,
      "indexType": "hnsw",
      "m": 16,
      "efConstruction": 200
    },
    "optimization": {
      "maxOutputTokens": 4000,
      "promptCompression": true,
      "dynamicTTL": {
        "baseSeconds": 3600,
        "tokenDecayFactor": 0.3,
        "minTTL": 300
      },
      "streamingPassthrough": true,
      "coalesceTimeoutMs": 3000
    },
    "monitoring": {
      "metricsEndpoint": "/metrics/ai-cache",
      "alertThresholds": {
        "hitRateMin": 0.60,
        "p95LatencyMaxMs": 450,
        "tokenSpendPer1kReqs": 1200000
      }
    }
  }
}

Quick Start Guide

1. Initialize Redis & Embedding Service: Deploy a Redis instance and configure an embedding provider. Set environment variables for REDIS_URL and EMBEDDING_API_KEY.
2. Instantiate the Optimizer: Import AICacheOptimizer, pass configuration, and call init(). Route all LLM calls through getOrGenerate(prompt).
3. Add Telemetry: Attach Prometheus/Grafana metrics to cache hit rate, latency percentiles, and token expenditure. Configure alerts for hit rate drops below 60%.
4. Validate Thresholds: Run a batch of 500 historical prompts through the optimizer. Adjust similarityThreshold until false positives stay below 5%. Deploy to production.