Model Routing Patterns for OpenAI-Compatible AI Gateways

plement the Model Registry

A registry centralizes model definitions and capabilities. This allows the router to validate requests and apply rules consistently.

export interface ModelDefinition {
    id: string;
    provider: string;
    capabilities: string[];
    costPerToken: number;
    isPremium: boolean;
}

class ModelRegistry {
    private models: Map<string, ModelDefinition> = new Map();

    register(model: ModelDefinition): void {
        this.models.set(model.id, model);
    }

    get(id: string): ModelDefinition | undefined {
        return this.models.get(id);
    }

    findByCapability(capability: string): ModelDefinition[] {
        return Array.from(this.models.values()).filter(m => 
            m.capabilities.includes(capability)
        );
    }
}

Step 3: Build the Intelligent Router

The router resolves the optimal model based on the context and registry rules. It prioritizes capability matching, then applies business logic for cost and locale.

class IntelligentRouter {
    constructor(private registry: ModelRegistry) {}

    resolve(context: RoutingContext): string {
        // Rule 1: Locale specialization
        if (context.locale === 'zh') {
            const zhModel = this.registry.findByCapability('chinese_nlp');
            if (zhModel.length > 0) return zhModel[0].id;
        }

        // Rule 2: Task-specific routing
        switch (context.taskType) {
            case 'reasoning':
                return this.selectPremiumModel('reasoning');
            case 'classification':
                return this.selectEconomyModel('classification');
            case 'multimodal':
                return this.selectModelByCapability('vision');
            default:
                return this.selectDefaultModel();
        }
    }

    private selectPremiumModel(capability: string): string {
        const candidates = this.registry.findByCapability(capability);
        return candidates.find(m => m.isPremium)?.id || candidates[0]?.id;
    }

    private selectEconomyModel(capability: string): string {
        const candidates = this.registry.findByCapability(capability);
        return candidates.find(m => !m.isPremium)?.id || candidates[0]?.id;
    }
}

Step 4: Resilient Execution with Fallback

Production systems must handle provider failures gracefully. The execution layer implements a fallback chain with circuit-breaking logic to prevent cascading latency.

import { OpenAI } from 'openai';

class ResilientExecutor {
    private client: OpenAI;
    private fallbackChain: string[];
    private maxRetries: number;

    constructor(client: OpenAI, config: { chain: string[]; maxRetries: number }) {
        this.client = client;
        this.fallbackChain = config.chain;
        this.maxRetries = config.maxRetries;
    }

    async execute(context: RoutingContext, prompt: string): Promise<CompletionResult> {
        const primaryModel = context.resolveModel(); // From router
        let attempts = 0;
        let currentModel = primaryModel;

        while (attempts <= this.maxRetries) {
            try {
                const response = await this.client.chat.completions.create({
                    model: currentModel,
                    messages: [{ role: 'user', content: prompt }],
                });
                
                this.emitMetrics(currentModel, 'success', response.usage);
                return response;
            } catch (error) {
                attempts++;
                this.emitMetrics(currentModel, 'error', error);
                
                if (attempts <= this.maxRetries && this.fallbackChain.length > attempts) {
                    currentModel = this.fallbackChain[attempts];
                    console.warn(`Fallback to ${currentModel} after error on ${currentModel}`);
                } else {
                    throw new Error(`Routing failed after ${attempts} attempts`);
                }
            }
        }
    }
}

Architecture Rationale:

• OpenAI Compatibility: Using the OpenAI SDK as the client interface standardizes the integration. The gateway translates requests to provider-specific formats, allowing the application to remain agnostic to backend differences.
• Registry Pattern: Centralizing model definitions prevents scattered configuration and enables runtime updates. New models can be added to the registry without code changes.
• Context-Aware Routing: Decisions are based on business context (locale, priority) rather than hardcoded strings. This supports complex requirements like bilingual workflows or SLA-based routing.
• Controlled Fallback: The executor limits retries and logs every fallback event. This prevents "fallback loops" that can mask reliability issues and inflate costs.

Pitfall Guide

Production AI systems face unique failure modes. The following pitfalls are derived from real-world deployment experience.

1. Blind Fallback Chains

   • Explanation: Implementing fallback chains without health checks or circuit breakers can lead to cascading failures. If multiple providers in the chain are degraded, the application may exhaust retries, increasing latency and cost without success.
   • Fix: Implement circuit breakers that track provider health. Skip providers that are currently rate-limited or returning high error rates. Use exponential backoff and jitter.

2. Cost-Per-Token Myopia

   • Explanation: Optimizing solely for token price ignores outcome quality. A cheaper model that hallucinates or requires multiple retries may have a higher total cost per successful action than a premium model.
   • Fix: Track "Cost per Successful Outcome." Correlate model choice with downstream metrics like user satisfaction, conversion rates, and support ticket volume. Route based on total cost of ownership, not just API price.

3. Static Routing Configurations

   • Explanation: Hardcoding routing rules in application code prevents dynamic optimization. Model performance and pricing change frequently; static rules become outdated quickly.
   • Fix: Externalize routing rules to a configuration service or database. Enable traffic splitting and A/B testing to compare models in production. Update routing weights based on real-time performance data.

4. Ignoring Latency in Fallback

   • Explanation: Fallback adds latency. If the primary model times out before triggering fallback, the user experience degrades significantly. Sequential fallbacks can compound this delay.
   • Fix: Set aggressive timeouts for the primary model. Consider speculative execution or parallel fallbacks for critical paths where latency is paramount. Monitor p99 latency per model and adjust timeouts dynamically.

5. SDK Contract Drift

   • Explanation: Assuming all providers perfectly match the OpenAI schema can lead to runtime errors. Providers may differ in parameter support, response formatting, or error codes.
   • Fix: Validate responses against expected schemas. Implement a normalization layer in the gateway or client to handle provider-specific quirks. Test routing rules against each provider's specific behavior.

6. Missing Routing Observability

   • Explanation: Without detailed metrics, teams cannot validate routing decisions. They may not know which models are failing, which are cost-effective, or how fallback impacts reliability.
   • Fix: Instrument every routing decision. Log model ID, latency, token usage, error codes, and fallback events. Build dashboards to track success rates, cost per workflow, and latency distributions by model.

7. Locale Blindness

   • Explanation: Routing all requests to a single English-centric model degrades quality for non-English workflows. Models like Qwen excel in Chinese processing, while others may struggle.
   • Fix: Include locale in the routing context. Define explicit rules for language-specific models. Validate model performance across languages before deployment.

Production Bundle

Action Checklist

• Audit Model Usage: Inventory all current AI calls, identifying task types, models, and costs.
• Define Routing Matrix: Map task types, locales, and priorities to target models.
• Deploy Gateway Client: Replace direct provider calls with the OpenAI-compatible gateway client.
• Implement Router: Deploy the IntelligentRouter with registry and context resolution.
• Configure Fallback: Set up fallback chains with max retries and circuit breakers.
• Instrument Metrics: Add logging for routing decisions, latency, errors, and token usage.
• Load Test Fallback: Simulate provider failures to verify fallback behavior and latency impact.
• Review Cost Metrics: Establish "Cost per Successful Outcome" tracking before optimizing for price.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-Stakes Reasoning	Premium Model (e.g., GPT-4o, Claude Sonnet 4)	Accuracy is critical; errors are costly.	High per-call, low total risk.
Bulk Classification	Economy Model (e.g., DeepSeek, Mini models)	Speed and cost are priorities; accuracy tolerance is higher.	Low per-call, high volume savings.
Chinese NLP Workflows	Regional Specialist (e.g., Qwen)	Superior quality and latency for Chinese text.	Medium; avoids rework costs.
Provider Outage	Fallback Chain	Maintains availability during upstream failures.	Variable; depends on fallback model pricing.
A/B Testing	Traffic Splitting	Validates model performance in production safely.	Neutral; controlled experiment cost.

Configuration Template

Use this YAML configuration to define routing rules and fallback behavior. This template supports dynamic updates without code changes.

routing:
  default_model: gpt-4o-mini
  rules:
    - condition: "context.locale == 'zh'"
      model: qwen-plus
      priority: 1
    - condition: "context.taskType == 'reasoning'"
      model: claude-sonnet-4
      priority: 2
    - condition: "context.taskType == 'classification'"
      model: deepseek-chat
      priority: 3
  fallback:
    chain:
      - gpt-4o
      - claude-sonnet-4
      - deepseek-chat
    max_retries: 2
    timeout_ms: 3000
    circuit_breaker:
      failure_threshold: 5
      recovery_timeout_ms: 60000
  metrics:
    track_cost_per_outcome: true
    log_routing_decisions: true
    alert_on_fallback_rate: 0.1

Quick Start Guide

1. Initialize Gateway Client: Install the OpenAI SDK and configure the client with your gateway API key and base URL.

   npm install openai
   

   const client = new OpenAI({
       apiKey: process.env.GATEWAY_API_KEY,
       baseURL: process.env.GATEWAY_BASE_URL,
   });
   

2. Define Routing Context: Create a context object for each request, specifying task type, locale, and priority.

   const ctx: RoutingContext = {
       taskId: 'req-123',
       taskType: 'reasoning',
       locale: 'en',
       priority: 'critical',
   };
   

3. Execute via Router: Pass the context to your router to resolve the model, then execute the request through the resilient executor.

   const model = router.resolve(ctx);
   const result = await executor.execute(ctx, "Analyze this data...");
   

4. Verify Observability: Check your metrics dashboard to confirm routing decisions, latency, and cost attribution are being recorded correctly. Adjust configuration as needed.