Implementation (TypeScript)

The following example demonstrates a modular inference orchestrator. It replaces direct API calls with a routed, cache-aware pipeline.

import { createHash } from 'crypto';
import { cosineSimilarity } from './vector-utils';
import { VectorStore } from './vector-store-client';
import { LLMProvider, ModelTier, RoutingConfig } from './types';

interface InferenceRequest {
  userId: string;
  query: string;
  context: Record<string, unknown>;
  priority: 'standard' | 'high';
}

interface CachedResponse {
  content: string;
  modelUsed: string;
  tokenCount: number;
  cachedAt: number;
}

export class InferenceOrchestrator {
  private vectorIndex: VectorStore;
  private provider: LLMProvider;
  private config: RoutingConfig;

  constructor(provider: LLMProvider, vectorIndex: VectorStore, config: RoutingConfig) {
    this.provider = provider;
    this.vectorIndex = vectorIndex;
    this.config = config;
  }

  async execute(request: InferenceRequest): Promise<CachedResponse> {
    // Step 1: Semantic interception
    const semanticHit = await this.checkSemanticCache(request.query);
    if (semanticHit) return semanticHit;

    // Step 2: Task classification & routing
    const targetModel = await this.routeRequest(request);
    
    // Step 3: Prompt composition with prefix caching
    const composedPrompt = this.composePrompt(request, targetModel);
    
    // Step 4: Bounded generation
    const response = await this.provider.generate(composedPrompt, {
      model: targetModel,
      maxTokens: this.config.outputLimits[targetModel],
      stopSequences: this.config.stopTokens,
      responseFormat: 'json_object'
    });

    // Step 5: Cache result for future semantic matches
    await this.indexSemanticResponse(request.query, response);

    return {
      content: response.text,
      modelUsed: targetModel,
      tokenCount: response.usage.total_tokens,
      cachedAt: Date.now()
    };
  }

  private async checkSemanticCache(query: string): Promise<CachedResponse | null> {
    const queryEmbedding = await this.provider.embed(query);
    const matches = await this.vectorIndex.search(queryEmbedding, { topK: 1 });
    
    if (matches.length > 0) {
      const similarity = cosineSimilarity(queryEmbedding, matches[0].embedding);
      if (similarity >= this.config.semanticThreshold) {
        return matches[0].metadata as CachedResponse;
      }
    }
    return null;
  }

  private async routeRequest(request: InferenceRequest): Promise<string> {
    const complexityScore = await this.provider.classifyComplexity(request.query);
    
    if (complexityScore < this.config.routingThresholds.low) {
      return ModelTier.ECONOMY;
    }
    if (complexityScore < this.config.routingThresholds.medium) {
      return ModelTier.STANDARD;
    }
    return ModelTier.FRONTER;
  }

  private composePrompt(request: InferenceRequest, model: string): string {
    const systemPrefix = this.getStableSystemPrompt(model);
    const dynamicContext = JSON.stringify(request.context);
    
    // Anthropic-style cache breakpoint injection
    return `${systemPrefix}\n---CACHE_BREAK---\n${dynamicContext}\nUser: ${request.query}`;
  }

  private getStableSystemPrompt(model: string): string {
    return this.config.systemPrompts[model] || this.config.systemPrompts.default;
  }

  private async indexSemanticResponse(query: string, response: any): Promise<void> {
    const embedding = await this.provider.embed(query);
    await this.vectorIndex.upsert({
      id: createHash('sha256').update(query).digest('hex'),
      vector: embedding,
      metadata: {
        content: response.text,
        modelUsed: response.model,
        tokenCount: response.usage.total_tokens,
        cachedAt: Date.now()
      } as CachedResponse
    });
  }
}

Why These Choices Matter

• Semantic Threshold at 0.90: A conservative 0.95 threshold misses reusable cache hits. An aggressive 0.85 risks serving subtly incorrect responses. 0.90 balances hit rate with accuracy, and can be dynamically adjusted based on user refinement signals.
• Explicit Output Constraints: Setting maxTokens, stopSequences, and responseFormat: 'json_object' prevents verbose scaffolding. Structured output typically reduces token consumption by 15-30% compared to natural language prose.
• Cache Breakpoint Injection: Placing a delimiter before dynamic context allows the provider to cache the stable system prefix. On Anthropic's API, cached reads cost 90% less ($0.03 vs $0.30 per million tokens for Claude 3 Haiku), with breakeven occurring after just two uses within the 5-minute TTL.
• Complexity-Based Routing: 60-80% of production requests (classification, extraction, short-form generation) do not require frontier capabilities. Routing to economy models like Claude 3 Haiku ($0.25/M input tokens) versus GPT-4o ($5-15/M input tokens) delivers immediate 40-70% spend reduction on routed traffic.

Pitfall Guide

1. Over-Aggressive Semantic Thresholds

Explanation: Setting cosine similarity below 0.85 causes the cache to serve responses for rephrased queries that carry different intent or constraints. This manifests as subtle hallucinations or outdated answers. Fix: Start at 0.90. Implement a feedback loop that tracks user refinement rates. If a cached response triggers immediate follow-up questions, log it as a low-quality hit and dynamically raise the threshold for that query cluster.

2. Caching Dynamic Prefixes

Explanation: Including user-specific variables, timestamps, or session IDs in the cached prefix breaks cache hit rates. The provider treats each variation as a new prefix, incurring write costs without read benefits. Fix: Strictly separate static system instructions from dynamic context. Use explicit breakpoints or delimiter patterns to ensure only immutable policy, role definitions, and formatting rules are cached.

3. Routing Complex Reasoning to Economy Models

Explanation: Classification models sometimes misjudge tasks requiring multi-step logic, chain-of-thought, or nuanced compliance checks. Economy models will generate plausible but incorrect outputs, increasing escalation costs. Fix: Implement a confidence score alongside complexity routing. If confidence falls below a configurable margin, force escalation to the standard or frontier tier. Log misrouted requests to retrain the classifier.

4. Ignoring Output Token Pricing

Explanation: Output tokens cost 3-4x more than inputs on most provider schedules (e.g., Claude 3.5 Sonnet charges $15/M for output vs $3/M for input). Unconstrained generation silently inflates bills. Fix: Enforce explicit length directives in system prompts. Mandate JSON mode for structured data extraction. Use stop sequences to terminate generation immediately after the required payload is produced.

5. Treating Quantization as a Free Optimization

Explanation: INT4 quantization can reduce VRAM requirements by 75%, but Amazon's February 2025 benchmarking showed a 39.46% accuracy drop on Llama-3.3 70B for certain workloads. INT8 is safer (0.5-2% degradation) but still requires validation. Fix: Never deploy quantized models to production without domain-specific evaluation suites. Run A/B tests against FP16 baselines. Reserve INT4 for narrow, high-volume tasks where accuracy tolerance is explicitly defined.

6. Batching Synchronous User Flows

Explanation: Batch APIs (OpenAI Batch, Anthropic Message Batches) offer 50% pricing but enforce a 24-hour completion window. Routing real-time chat or transactional flows to batch endpoints breaks user experience. Fix: Strictly segregate workloads. Use batch routing only for offline ETL, nightly classification, dataset enrichment, and evaluation suites. Maintain separate pipeline handlers for synchronous vs asynchronous traffic.

7. OSS Model Sprawl for General Tasks

Explanation: Fine-tuned open-source models (e.g., Llama-3 8B on an A10G) can handle ~500 requests/minute at ~$0.0002/request, a 25-75x cost advantage over frontier APIs. However, deploying them for broad, open-ended tasks causes quality degradation that triggers rework. Fix: Limit OSS deployment to narrow, well-defined tasks: sentiment analysis, NER, document classification, and common language translation. Maintain frontier models for open-ended reasoning and complex instruction following.

Production Bundle

Action Checklist

• Instrument current API spend: Tag all requests by model, token type, and endpoint to establish a baseline cost per successful request.
• Deploy semantic cache layer: Integrate an embedding model and vector store, set initial cosine threshold to 0.90, and route cache hits before LLM dispatch.
• Configure model routing: Implement a lightweight classifier to score task complexity, define routing thresholds, and enable fallback escalation on low confidence.
• Enable prefix caching: Audit system prompts for static content, inject provider-specific cache breakpoints, and verify TTL hit rates in provider dashboards.
• Enforce output constraints: Add explicit length directives, mandate JSON mode for structured outputs, and configure stop sequences to prevent token bloat.
• Segregate batch workloads: Identify offline pipelines, migrate them to batch endpoints, and verify 24-hour SLA compliance.
• Establish quality monitoring: Track user refinement rates, escalation frequency, and cache hit accuracy to dynamically adjust thresholds.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time conversational UI	Semantic Cache + Model Routing	Maintains sub-second latency while intercepting ~30% of repeat queries and routing simple intents to economy models	-45% to -60%
Async document processing	Batch Inference + Context Compression	24-hour window is acceptable; rerankers reduce RAG context by 50-70%, lowering input token volume	-50% (batch) + -30% (compression)
High-accuracy compliance checks	Frontier Routing + Output Structuring	Complex reasoning requires frontier capabilities; JSON mode and strict length limits prevent token waste	-15% to -25%
High-volume classification	OSS Fine-Tuned Model + Semantic Cache	Narrow task fits OSS strengths; cache eliminates redundant compute on repeat categories	-70% to -85%
Multi-turn customer support	Prefix Caching + Conversation Summarization	Stable policy prompts benefit from 90% cheaper cached reads; summarization caps context window growth	-40% to -55%

Configuration Template

// inference.config.ts
export const routingConfig = {
  semanticThreshold: 0.90,
  routingThresholds: {
    low: 0.35,
    medium: 0.65
  },
  outputLimits: {
    'claude-3-haiku': 512,
    'claude-3-5-sonnet': 1024,
    'gpt-4o': 1024
  },
  stopTokens: ['\n\n', '```', 'END_RESPONSE'],
  systemPrompts: {
    'claude-3-haiku': `You are a precise extraction assistant. Return only JSON.`,
    'claude-3-5-sonnet': `You are a reasoning engine. Follow step-by-step logic.`,
    'default': `You are a helpful assistant.`
  },
  cacheBreakpoint: '---CACHE_BREAK---',
  batchWindow: '24h',
  ossTaskWhitelist: ['sentiment', 'ner', 'classification', 'translation']
};

Quick Start Guide

1. Install dependencies: Add your preferred vector store client, embedding model SDK, and LLM provider wrapper to your project.
2. Initialize the orchestrator: Instantiate InferenceOrchestrator with your provider, vector index, and the configuration template above.
3. Replace direct API calls: Swap synchronous provider.generate() invocations with orchestrator.execute(request). Ensure all requests include query, context, and priority fields.
4. Verify cache hits: Monitor your vector store metrics and provider dashboard. Confirm that prefix cache read rates exceed 60% and semantic cache intercepts 25-35% of traffic within the first 48 hours.
5. Tune thresholds: Adjust semanticThreshold and routingThresholds based on refinement rates and escalation logs. Deploy changes incrementally to avoid sudden quality shifts.