pricing-rules-v1.yaml

o infrastructure cost. Input tokens, output tokens, caching hits, model routing, and regional pricing all affect the actual compute cost. Build a cost mapping service that translates request metadata into a base cost.

// cost-mapper.ts
export interface RequestMetadata {
  model: string;
  inputTokens: number;
  outputTokens: number;
  cacheHit: boolean;
  region: string;
}

export interface CostMapping {
  baseCostPerInputToken: number;
  baseCostPerOutputToken: number;
  cacheMultiplier: number;
  regionalSurcharge: number;
}

export class CostMapper {
  private mappings: Record<string, CostMapping> = {
    'gpt-4o': {
      baseCostPerInputToken: 0.0000025,
      baseCostPerOutputToken: 0.00001,
      cacheMultiplier: 0.3,
      regionalSurcharge: 1.0
    },
    'claude-3-sonnet': {
      baseCostPerInputToken: 0.000003,
      baseCostPerOutputToken: 0.000015,
      cacheMultiplier: 0.25,
      regionalSurcharge: 1.05
    }
  };

  calculateCost(metadata: RequestMetadata): number {
    const mapping = this.mappings[metadata.model] ?? this.mappings['gpt-4o'];
    const inputCost = metadata.inputTokens * mapping.baseCostPerInputToken;
    const outputCost = metadata.outputTokens * mapping.baseCostPerOutputToken;
    const cacheDiscount = metadata.cacheHit ? mapping.cacheMultiplier : 1.0;
    const regionalFactor = mapping.regionalSurcharge;

    return (inputCost + (outputCost * cacheDiscount)) * regionalFactor;
  }
}

Step 2: Real-Time Metering Pipeline

Metering must be idempotent, event-driven, and decoupled from the inference path. Use a message queue or event bus to capture request completion events. Assign idempotency keys to prevent double-counting during retries or network partitions.

// metering-pipeline.ts
import { randomUUID } from 'crypto';

export interface MeteringEvent {
  idempotencyKey: string;
  customerId: string;
  featureId: string;
  metadata: RequestMetadata;
  baseCost: number;
  timestamp: number;
}

export class MeteringPipeline {
  private processedKeys = new Set<string>();
  private eventQueue: MeteringEvent[] = [];

  async ingest(event: Omit<MeteringEvent, 'idempotencyKey'>): Promise<void> {
    const idempotencyKey = `${event.customerId}:${event.featureId}:${event.timestamp}:${randomUUID()}`;
    
    if (this.processedKeys.has(idempotencyKey)) return;
    
    const fullEvent: MeteringEvent = { ...event, idempotencyKey };
    this.eventQueue.push(fullEvent);
    this.processedKeys.add(idempotencyKey);
    
    // Persist to event store / message broker
    await this.publishToBroker(fullEvent);
  }

  private async publishToBroker(event: MeteringEvent): Promise<void> {
    // Integration with Kafka, SQS, or Redis Streams
    console.log(`[METER] Published ${event.featureId} for ${event.customerId}`);
  }
}

Step 3: Pricing Rules Engine

Pricing rules must be versioned, testable, and evaluated against metered events. Support tiered pricing, volume discounts, and cost-plus markups with guardrails to prevent margin erosion during cost spikes.

// pricing-engine.ts
export interface PricingRule {
  version: string;
  markupPercentage: number;
  minMarginFloor: number;
  volumeDiscountThresholds: { tokens: number; discount: number }[];
  costSpikeGuardrail: { maxCostPerRequest: number; action: 'cap' | 'alert' | 'allow' };
}

export class PricingEngine {
  private activeRule: PricingRule;

  constructor(initialRule: PricingRule) {
    this.activeRule = initialRule;
  }

  calculateCustomerCost(baseCost: number, tokens: number): number {
    let cost = baseCost * (1 + this.activeRule.markupPercentage / 100);

    // Volume discount
    for (const tier of this.activeRule.volumeDiscountThresholds) {
      if (tokens >= tier.tokens) {
        cost *= (1 - tier.discount / 100);
        break;
      }
    }

    // Margin floor enforcement
    const margin = (cost - baseCost) / cost;
    if (margin < this.activeRule.minMarginFloor / 100) {
      cost = baseCost / (1 - this.activeRule.minMarginFloor / 100);
    }

    // Cost spike guardrail
    if (cost > this.activeRule.costSpikeGuardrail.maxCostPerRequest) {
      if (this.activeRule.costSpikeGuardrail.action === 'cap') {
        cost = this.activeRule.costSpikeGuardrail.maxCostPerRequest;
      } else if (this.activeRule.costSpikeGuardrail.action === 'alert') {
        console.warn(`[PRICING] Cost spike detected: ${cost}`);
      }
    }

    return Math.round(cost * 10000) / 10000; // 4 decimal precision
  }

  updateRule(newRule: PricingRule): void {
    this.activeRule = newRule;
  }
}

Step 4: Billing Sync & Customer Visibility

Sync metered and priced events to your billing provider (Stripe, Chargebee, etc.) using a deduplicated, idempotent API. Expose real-time usage dashboards to customers to prevent bill shock. Cache pricing results per session to reduce compute overhead, but invalidate on rule version changes.

Architecture Decisions & Rationale:

• Event-Driven Metering: Decouples inference latency from pricing logic. Ensures accurate accounting even during retries or partial failures.
• Idempotency Keys: Prevents double-billing during network partitions or SDK retries. Critical for financial auditability.
• Rule Versioning: Pricing changes must be backward-compatible. Store rule versions in a configuration service; apply version at event ingestion time.
• Cost Caching: Repeated embeddings or system prompts can be cached. Cache hits reduce base cost by 30–60%, directly improving margin.
• Guardrails Over Hard Caps: Hard caps create arbitrage opportunities. Guardrails with alerts and dynamic markups preserve margins while maintaining customer trust.

Pitfall Guide

1. Treating Tokens as a Direct Cost Proxy

Tokens are a billing unit, not a compute metric. Actual cost depends on model architecture, context window utilization, and provider infrastructure pricing. A 10k-token request to a lightweight model may cost less than a 2k-token request to a reasoning model. Map tokens to actual provider invoices, not theoretical multipliers.

2. Hardcoding Pricing Tiers Without Volatility Buffers

Static tiers assume stable inference costs. AI provider pricing changes quarterly. Regional surcharges, cache misses, and output variance break static math. Implement dynamic markups with margin floors and cost spike detection. Update pricing rules independently of application deployments.

3. Missing Idempotency in Metering Events

Retries, SDK timeouts, and load balancer reconnections generate duplicate events. Without idempotency keys, your billing provider charges twice. Store processed keys in a distributed cache with TTL matching your billing window. Reject duplicates before cost calculation.

4. Ignoring Prompt Engineering Variance

System prompts, few-shot examples, and conversation history inflate input tokens unpredictably. Customers who paste large contexts or use verbose prompts drive up costs without proportional value. Segment metering by prompt type. Apply different pricing rules for system vs user tokens.

5. No Real-Time Cost Visibility for Customers

Bill shock is the primary driver of AI feature churn. Customers cannot optimize usage they cannot see. Expose a usage dashboard with token breakdown, cost per request, and projected monthly spend. Update it via webhooks or polling. Transparency reduces support tickets and cancellation rates.

6. Over-Engineering the Pricing Engine

Building a monolithic pricing service that handles inference, metering, and billing sync creates coupling and latency. Separate concerns: inference service emits events, metering pipeline ingests, pricing engine evaluates, billing sync publishes. Use event sourcing for auditability. Keep the pricing engine stateless and rule-driven.

7. Skipping Cost Caching for Repeated Embeddings/Prompts

Vector embeddings and system prompts are often identical across requests. Recalculating or re-sending them to providers wastes compute. Cache embeddings at the application layer. Track cache hit rates in metering events. Apply cache discounts automatically in the pricing engine.

Best Practices from Production:

• Version pricing rules in a dedicated configuration service. Roll back instantly on margin anomalies.
• Implement circuit breakers for cost spikes. Alert engineering and product when base cost exceeds thresholds.
• Use 4-decimal precision for cost calculations. Rounding errors compound across millions of requests.
• Expose pricing rule impact reports to product teams. Correlate pricing changes with adoption and churn.
• Run pricing simulations against historical metering data before deploying rule updates.

Production Bundle

Action Checklist

• Implement idempotent metering pipeline with distributed key tracking
• Build cost mapper that translates provider invoices to request-level base costs
• Deploy versioned pricing engine with margin floors and spike guardrails
• Sync metered events to billing provider using idempotent API calls
• Expose real-time usage dashboard with token breakdown and projected spend
• Cache embeddings and system prompts; track cache hit rates in metering
• Run pricing simulation against 30 days of historical events before rule deployment
• Document pricing rule versioning strategy and rollback procedures

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Low-volume internal AI tool	Static usage-based with flat markup	Simplicity outweighs marginal variance	±8% margin fluctuation
Public-facing AI SaaS	Dynamic cost-plus with guardrails	Predictable margins, low churn, audit-ready	±3% margin fluctuation
Enterprise AI with volume discounts	Tiered dynamic pricing with cost caching	Aligns pricing with actual compute savings	12–18% margin improvement
Experimental AI feature launch	Flat-rate with cost monitoring	Accelerates adoption while collecting telemetry	Temporary margin subsidy, data-driven pivot

Configuration Template

# pricing-rules-v1.yaml
version: "v1.2.0"
effective_date: "2024-06-01T00:00:00Z"

markup:
  base_percentage: 45
  min_margin_floor: 28

volume_discounts:
  - tokens: 100000
    discount: 5
  - tokens: 500000
    discount: 12
  - tokens: 1000000
    discount: 18

guardrails:
  cost_spike:
    max_cost_per_request_usd: 0.85
    action: "alert" # cap | alert | allow
  cache_discount:
    enabled: true
    multiplier: 0.3

billing_sync:
  provider: "stripe"
  idempotency_window_hours: 72
  decimal_precision: 4

Quick Start Guide

1. Instrument Inference Endpoints: Add a middleware that captures request metadata (model, tokens, cache hit, region) and emits a metering event to your message broker. Include an idempotency key derived from customer ID, feature ID, and timestamp.
2. Deploy Cost Mapper & Pricing Engine: Load the configuration template into your pricing service. Wire the cost mapper to your provider invoice data. Connect the pricing engine to consume metering events and calculate customer cost.
3. Sync to Billing Provider: Implement a deduplicated publisher that sends priced events to Stripe/Chargebee. Use idempotency keys and handle 409/429 responses with exponential backoff.
4. Expose Usage Dashboard: Build a read-only API that aggregates metered events by customer. Return token breakdown, base cost, markup, and final customer cost. Update via webhook or 60-second polling.
5. Validate with Simulation: Run the pricing engine against 30 days of historical metering data. Compare calculated revenue against actual provider invoices. Adjust markup and guardrails until margin variance falls below 5%. Deploy rule version. Monitor for 72 hours. Roll back if anomalies exceed thresholds.