en.get_encoding("cl100k_base")

async def dispatch(self, request: Request, call_next):
    # Extract tenant context from headers/jwt
    tenant_id = request.headers.get("X-Tenant-ID", "anonymous")
    feature_id = request.url.path.split("/")[2] if len(request.url.path.split("/")) > 2 else "unknown"
    
    # Count input tokens
    body = await request.json()
    prompt = body.get("prompt", "")
    input_tokens = len(self.encoding.encode(prompt))
    
    # Call downstream AI service
    response = await call_next(request)
    
    # Parse response body for output tokens
    response_body = b""
    async for chunk in response.body_iterator:
        response_body += chunk
    
    # Decode and count output tokens
    try:
        output_text = response_body.decode("utf-8")
        output_tokens = len(self.encoding.encode(output_text))
    except Exception:
        output_tokens = 0
        
    # Build metering event
    event = {
        "tenant_id": tenant_id,
        "feature_id": feature_id,
        "model": request.headers.get("X-AI-Model", "default"),
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "latency_ms": response.headers.get("X-Response-Time", 0),
        "timestamp": asyncio.get_event_loop().time()
    }
    
    # Push to async queue for billing worker
    await self.metering_queue.put(event)
    
    # Reconstruct response for client
    response = Response(content=response_body, status_code=response.status_code)
    for key, value in response.headers.items():
        response.headers[key] = value
    return response

**Key Engineering Considerations:**
- `tiktoken` is used for accurate tokenization matching provider behavior.
- Response body reconstruction prevents double-consumption errors in FastAPI.
- Queue-based emission ensures zero impact on inference latency.
- Idempotency keys should be added in production to handle retry storms.

### Architecture Decisions

| Decision | Recommendation | Rationale |
|----------|----------------|-----------|
| Metering Timing | At inference boundary | Captures actual compute, excludes client-side retries |
| Event Delivery | Async queue + idempotent consumers | Prevents billing latency from blocking AI responses |
| Cost Attribution | Tenant + Feature + Model Version | Enables granular margin analysis and dynamic pricing |
| Streaming Support | Chunk-level token estimation or provider metadata | Real-time counting in streaming requires stateful parsers |
| Fallback Routing | Cost-aware model router | Routes to cheaper models when spend thresholds are approached |
| Cache Handling | Exclude cache hits from billing | Rewards optimization, reduces customer disputes |

Production systems should deploy a dedicated `BillingWorker` service that consumes metering events, applies pricing rules, and syncs with Stripe, Chargebee, or custom ledger systems. Reconciliation jobs must run nightly to detect token count drift between internal metering and provider invoices.

---

## Pitfall Guide

1. **Counting Only Response Tokens**
   Input tokens often consume 60–80% of inference cost, especially in RAG pipelines. Ignoring prompt length guarantees margin leakage.

2. **Hardcoding Pricing Tiers**
   Provider rates change quarterly. Static pricing requires manual updates and creates billing mismatches. Use multiplier-based pricing tied to a live rate card.

3. **Synchronous Billing Calls**
   Blocking inference while waiting for billing confirmation increases p99 latency by 200–400ms. Always decouple metering from response delivery.

4. **Ignoring Model Version Drift**
   Newer models may be faster but cost 3x more. Without version-tagged metering, you cannot attribute cost spikes or adjust pricing dynamically.

5. **No Cost-Cap or Fallback Mechanism**
   Runaway context windows or retry loops can generate $500+ in minutes. Implement tenant spend limits with automatic routing to cheaper models or cached responses.

6. **Billing Latency > 24 Hours**
   Users expect real-time usage visibility. Delayed billing correlates with high support volume and payment failures. Push usage dashboards via WebSockets or SSE.

7. **Overlooking Cache & Optimization Discounts**
   If your system caches embeddings or reuses completions, billing should reflect actual compute consumed, not theoretical maximums. Otherwise, power users will churn.

---

## Production Bundle

### Action Checklist
- [ ] Map all AI endpoints to tenant, feature, and model version identifiers
- [ ] Implement token counting at the inference boundary using provider-matched encoders
- [ ] Deploy async metering queue with idempotent consumer workers
- [ ] Configure tenant-level spending caps and cost-aware fallback routing
- [ ] Integrate metering events with billing provider via webhooks or ledger API
- [ ] Build real-time usage dashboard with WebSocket/SSE push
- [ ] Schedule nightly reconciliation jobs comparing internal tokens vs provider invoices
- [ ] Document pricing multipliers and review quarterly against provider rate cards

### Decision Matrix

| Product Stage | Cost Volatility | User Base Size | Engineering Bandwidth | Recommended Model |
|---------------|-----------------|----------------|------------------------|-------------------|
| MVP / Internal | High | < 50 | Low | Flat + Manual Review |
| Growth / Beta | Medium | 50–5,000 | Medium | Tiered + Usage Alerts |
| Scale / Public | High | > 5,000 | High | Hybrid (Base + Token) |
| Enterprise | Low/Medium | Custom | Dedicated | Contract + Metered Reconciliation |

### Configuration Template

```yaml
pricing:
  currency: USD
  rounding: nearest_cent
  rules:
    - model: "gpt-4o"
      input_rate: 0.000005
      output_rate: 0.000015
      multiplier: 1.0
    - model: "claude-3-5-sonnet"
      input_rate: 0.000003
      output_rate: 0.000015
      multiplier: 1.0
    - model: "fallback-llama-3"
      input_rate: 0.0000005
      output_rate: 0.000002
      multiplier: 0.8

limits:
  tenant:
    daily_cap: 50.00
    burst_threshold: 0.8
    fallback_model: "fallback-llama-3"
  feature:
    max_context_tokens: 128000
    cache_ttl_seconds: 3600
    exclude_cache_from_billing: true

metering:
  queue: "ai-metering-events"
  consumer_concurrency: 8
  idempotency_window_hours: 24
  reconciliation_schedule: "0 2 * * *"

Quick Start Guide

1. Instrument Endpoints: Wrap AI inference calls with a metering decorator that captures tenant ID, model version, and token counts using a provider-matched encoder.
2. Deploy Event Pipeline: Configure a message queue (SQS/Kafka) and a consumer worker that applies pricing rules from the configuration template, calculates cost, and pushes to your billing ledger.
3. Enforce Safeguards: Add middleware that checks tenant spend against daily caps. When thresholds are breached, route to fallback models or enable response caching.
4. Expose Usage Visibility: Build a lightweight dashboard that subscribes to metering events via WebSocket. Display real-time token consumption, estimated cost, and remaining cap.
5. Reconcile & Iterate: Run nightly jobs comparing internal metering against provider invoices. Adjust pricing multipliers quarterly. Deprecate models that show >15% cost drift without margin compensation.

AI feature pricing is not a billing exercise; it is a distributed telemetry problem. When engineered correctly, it transforms volatile compute costs into predictable margins, aligns customer expectations with actual resource consumption, and provides the data foundation for model optimization and fallback routing. Treat metering as first-class infrastructure, and your AI features will scale without margin erosion or user friction.