interface RoutingInput {
  query: string;
  availableRoutes: string[];
}

interface RoutingOutput {
  selectedRoute: string;
  rawConfidence: number;
  calibratedConfidence: number;
  action: 'execute' | 'abstain' | 'escalate';
}

class ConfidenceExtractor {
  private readonly apiClient: LLMClient;
  private readonly sampleCount: number;

  constructor(client: LLMClient, samples: number = 5) {
    this.apiClient = client;
    this.sampleCount = samples;
  }

  async extract(input: RoutingInput): Promise<{ vote: string; fraction: number }> {
    const prompts = Array.from({ length: this.sampleCount }, () => 
      this.buildRoutingPrompt(input)
    );

    const responses = await Promise.all(
      prompts.map(p => this.apiClient.generate(p, { temperature: 0.7 }))
    );

    const votes = responses.map(r => r.trim().toLowerCase());
    const voteCounts = votes.reduce<Record<string, number>>((acc, v) => {
      acc[v] = (acc[v] || 0) + 1;
      return acc;
    }, {});

    const dominantRoute = Object.entries(voteCounts).sort((a, b) => b[1] - a[1])[0][0];
    const fraction = voteCounts[dominantRoute] / this.sampleCount;

    return { vote: dominantRoute, fraction };
  }

  private buildRoutingPrompt(input: RoutingInput): string {
    return `Route the following query to exactly one of these options: ${input.availableRoutes.join(', ')}.
Query: "${input.query}"
Return only the route name.`;
  }
}

class ProbabilityCalibrator {
  private readonly weight: number;
  private readonly bias: number;

  constructor(weight: number, bias: number) {
    this.weight = weight;
    this.bias = bias;
  }

  static fromCalibrationSet(rawScores: number[], trueLabels: boolean[]): ProbabilityCalibrator {
    // In production, use scipy.optimize or a JS equivalent

to minimize log-loss // Here we simulate fitted parameters from a 1000-sample calibration set return new ProbabilityCalibrator(2.14, -1.87); }

calibrate(rawFraction: number): number { const z = this.weight * rawFraction + this.bias; return 1 / (1 + Math.exp(-z)); } }

**Architecture Rationale:** Platt scaling requires only two parameters, making it stable with 200–500 calibration examples. It outperforms histogram binning in smoothness and avoids the overfitting risks of isotonic regression on smaller datasets.

### Step 3: Enforce Thresholds & Abstention Zones

Calibrated probabilities enable cost-aware routing. Symmetric errors use a 0.5 cutoff; asymmetric costs shift the threshold. An abstention zone handles irreducible ambiguity.

```typescript
class DecisionRouter {
  private readonly extractor: ConfidenceExtractor;
  private readonly calibrator: ProbabilityCalibrator;
  private readonly thresholds: { execute: number; abstain: number };

  constructor(
    extractor: ConfidenceExtractor,
    calibrator: ProbabilityCalibrator,
    thresholds: { execute: number; abstain: number }
  ) {
    this.extractor = extractor;
    this.calibrator = calibrator;
    this.thresholds = thresholds;
  }

  async route(input: RoutingInput): Promise<RoutingOutput> {
    const { vote, fraction } = await this.extractor.extract(input);
    const calibrated = this.calibrator.calibrate(fraction);

    let action: RoutingOutput['action'];
    if (calibrated >= this.thresholds.execute) {
      action = 'execute';
    } else if (calibrated >= this.thresholds.abstain) {
      action = 'abstain';
    } else {
      action = 'escalate';
    }

    return {
      selectedRoute: vote,
      rawConfidence: fraction,
      calibratedConfidence: calibrated,
      action
    };
  }
}

Architecture Rationale: The three-tier action space (execute, abstain, escalate) decouples model uncertainty from business logic. Abstention triggers clarifying prompts or parallel execution; escalation routes to human review or deterministic fallbacks.

Step 4: Validate Calibration Quality

Production systems must continuously verify calibration using Expected Calibration Error (ECE) and Brier scores.

function computeECE(predictions: number[], labels: boolean[], bins: number = 10): number {
  const binCounts = new Array(bins).fill(0);
  const binAccuracy = new Array(bins).fill(0);
  const binConfidence = new Array(bins).fill(0);

  predictions.forEach((p, i) => {
    const binIndex = Math.min(Math.floor(p * bins), bins - 1);
    binCounts[binIndex]++;
    binAccuracy[binIndex] += labels[i] ? 1 : 0;
    binConfidence[binIndex] += p;
  });

  let ece = 0;
  const total = predictions.length;
  for (let i = 0; i < bins; i++) {
    if (binCounts[i] === 0) continue;
    const avgConf = binConfidence[i] / binCounts[i];
    const avgAcc = binAccuracy[i] / binCounts[i];
    ece += (binCounts[i] / total) * Math.abs(avgAcc - avgConf);
  }
  return ece;
}

Architecture Rationale: ECE measures the weighted average gap between predicted confidence and empirical accuracy per bin. Values below 0.05 indicate production-ready calibration. Pair this with reliability diagrams to visualize drift over time.

Pitfall Guide

1. The Self-Report Trap

Explanation: Prompting an LLM to output a confidence score yields systematically inflated values. Models lack true meta-cognition and default to high certainty to satisfy instruction-following biases. Fix: Never trust self-reported probabilities. Extract signals from token distributions, self-consistency voting, or cross-model disagreement, then apply Platt scaling.

2. Static Thresholds in Shifting Traffic

Explanation: Thresholds tuned on Q1 traffic degrade when user intent distribution changes. A fixed 0.6 cutoff may become too aggressive or too conservative as query patterns evolve. Fix: Implement rolling-window calibration. Recalculate Platt parameters monthly using the most recent 500 labeled examples. Alert when ECE exceeds 0.08.

3. Ignoring Cost Asymmetry

Explanation: Routing a knowledge query to web search wastes tokens and latency. Routing a search-dependent query to local knowledge produces factual errors. These mistakes carry different business costs. Fix: Weight threshold tuning by cost ratios. If false negatives cost 5× more than false positives, shift the execute threshold downward and widen the abstention zone.

4. The Gold Set Mirage

Explanation: Synthetic benchmarks or manually crafted examples rarely match production traffic distributions. Models optimized on clean data fail on noisy, real-world inputs. Fix: Stratify your evaluation set using production log sampling. Match the class distribution, query length, and ambiguity rate of live traffic.

5. Calibration Drift on Model Updates

Explanation: Vendors frequently push silent model updates. A calibration curve fitted on model-v2.1 becomes invalid on model-v2.2, causing ECE to spike without warning. Fix: Version-lock calibration datasets. Run automated calibration regression tests against a shadow deployment before promoting new model versions.

6. Overfitting Calibration Curves

Explanation: Isotonic regression and histogram binning require large datasets. Applying them to 200 examples creates jagged, non-generalizable mappings that fail in production. Fix: Start with Platt scaling. Only upgrade to non-parametric methods when you have 1,000+ calibration examples and Platt's logistic assumption proves insufficient.

7. Treating Ambiguity as Failure

Explanation: Some inputs are inherently unclear. Forcing a decision on ambiguous queries increases error rates and degrades user trust. Fix: Design abstention zones as first-class features. Trigger clarifying questions, parallel tool execution, or human-in-the-loop review when calibrated confidence falls in the uncertainty band.

Production Bundle

Action Checklist

• Build a stratified gold set from production logs matching live traffic distribution
• Implement self-consistency sampling (N=5) as the primary confidence signal
• Fit Platt scaling parameters on a held-out calibration set (min. 300 examples)
• Configure asymmetric thresholds based on business cost ratios
• Deploy ECE monitoring with automated alerts when calibration error exceeds 0.08
• Version-lock calibration datasets and run shadow tests before model updates
• Implement abstention routing with clarifying prompts or parallel execution fallbacks

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Low-latency internal routing	Platt-scaled self-consistency (N=3)	Balances speed and calibration quality	Moderate (+15% compute)
High-stakes customer triage	Cross-model ensemble + isotonic regression	Maximizes calibration accuracy for critical decisions	High (+40% compute)
Ambiguous query volume >20%	Wide abstention zone (0.35–0.65)	Prevents forced errors on unclear inputs	Low (shifts cost to clarification)
Vendor model updates frequent	Rolling-window Platt + shadow deployment	Detects calibration drift before production impact	Low (automated monitoring)

Configuration Template

export const routingConfig = {
  sampling: {
    strategy: 'self_consistency',
    count: 5,
    temperature: 0.7,
    maxConcurrency: 10
  },
  calibration: {
    method: 'platt_scaling',
    weight: 2.14,
    bias: -1.87,
    updateFrequency: 'monthly',
    minCalibrationSamples: 300
  },
  thresholds: {
    execute: 0.68,
    abstain: 0.42,
    costWeighting: { falsePositive: 1, falseNegative: 4 }
  },
  monitoring: {
    eceWarning: 0.08,
    eceCritical: 0.12,
    reliabilityDiagramExport: true,
    driftDetectionWindow: '14d'
  },
  fallback: {
    abstentionAction: 'clarify_prompt',
    escalationAction: 'human_review_queue',
    timeoutMs: 2000
  }
};

Quick Start Guide

1. Extract live traffic samples: Pull 1,000 recent user queries from your application logs. Stratify by intent category to match production distribution.
2. Label a gold set: Route 300 samples through your target LLM using self-consistency (N=5). Have two annotators label the correct route. Calculate inter-annotator agreement to establish your Bayes error floor.
3. Fit calibration parameters: Split the labeled set into 70% calibration and 30% validation. Run Platt scaling optimization to derive weight and bias parameters. Verify ECE < 0.05 on the validation split.
4. Deploy with abstention: Initialize the DecisionRouter with calibrated parameters and cost-weighted thresholds. Route production traffic through the abstention zone for the first 48 hours, logging all escalations for threshold refinement.
5. Monitor drift: Configure ECE tracking and reliability diagram exports. Set automated alerts for calibration degradation. Schedule monthly Platt parameter retraining using the most recent labeled cohort.