ai-pmf-config.yaml

n follows a five-step architecture.

Step 1: Define Outcome-First Telemetry Schema

Model metrics cannot proxy for product value. Replace accuracy tracking with outcome completion events. The telemetry schema must capture:

• trigger_context: User action that initiated the AI feature
• model_output: Raw or processed model response
• outcome_action: User behavior following the output (accept, edit, discard, retry, ignore)
• latency_budget_ms: P95 threshold for the workflow
• cost_per_token: Current routing cost

Step 2: Implement Shadow Mode Evaluation

Deploy the AI component in shadow mode before exposing it to users. Shadow mode runs the model pipeline alongside the existing deterministic logic, logs outputs, and compares them against historical user behavior. This isolates model drift and validates outcome correlation without risking production stability.

Step 3: Build Deterministic Fallback & Cost-Aware Routing

AI outputs must never block critical workflows. Implement a routing layer that:

• Routes to lightweight models for high-frequency, low-complexity tasks
• Falls back to deterministic rules or cached responses when latency exceeds the budget
• Logs fallback triggers as outcome signals (fallbacks indicate model failure to meet workflow thresholds)

Step 4: Calculate Behavioral PMF Metrics

Track three core metrics:

• Outcome Completion Rate (OCR): Percentage of AI interactions resulting in accepted output
• Workflow Friction Index (WFI): Ratio of retries/edits to total interactions
• Cost-per-Value (CPV): Operational cost divided by successful outcome completions

Step 5: Iterate via Continuous Evaluation Pipeline

Feed behavioral metrics back into the model selection and prompt orchestration pipeline. Retrain or swap models only when CPV increases or OCR drops below the workflow threshold. Static evaluation datasets must be refreshed quarterly with production outcome logs.

TypeScript Implementation: Telemetry & Shadow Mode Router

// telemetry.ts
export interface AIOutcomeEvent {
  featureId: string;
  triggerContext: string;
  modelOutput: string;
  outcomeAction: 'accepted' | 'edited' | 'discarded' | 'retried' | 'ignored';
  latencyMs: number;
  costCents: number;
  timestamp: number;
}

export class OutcomeTelemetry {
  private events: AIOutcomeEvent[] = [];

  track(event: AIOutcomeEvent): void {
    this.events.push({ ...event, timestamp: Date.now() });
    // In production: batch send to event pipeline (Kafka, PostHog, Segment)
  }

  getOutcomeCompletionRate(featureId: string): number {
    const featureEvents = this.events.filter(e => e.featureId === featureId);
    if (featureEvents.length === 0) return 0;
    const accepted = featureEvents.filter(e => e.outcomeAction === 'accepted').length;
    return accepted / featureEvents.length;
  }
}

// router.ts
import { OutcomeTelemetry } from './telemetry';

export interface ModelRoute {
  provider: string;
  model: string;
  latencyBudgetMs: number;
  fallbackRule: () => Promise<string>;
}

export class AIRouter {
  constructor(
    private route: ModelRoute,
    private telemetry: OutcomeTelemetry
  ) {}

  async execute(prompt: string, featureId: string): Promise<string> {
    const start = performance.now();
    
    try {
      // Simulate model call with timeout enforcement
      const modelOutput = await Promise.race([
        this.callModel(prompt),
        new Promise<string>((_, reject) => 
          setTimeout(() => reject(new Error('LATENCY_TIMEOUT')), this.route.latencyBudgetMs)
        )
      ]);

      const latency = performance.now() - start;
      this.telemetry.track({
        featureId,
        triggerContext: prompt.slice(0, 100),
        modelOutput,
        outcomeAction: 'accepted', // Updated via UI interaction hook
        latencyMs: latency,
        costCents: this.calculateCost(prompt.length),
      });

      return modelOutput;
    } catch (error) {
      const fallback = await this.route.fallbackRule();
      this.telemetry.track({
        featureId,
        triggerContext: prompt.slice(0, 100),
        modelOutput: '[FALLBACK]',
        outcomeAction: 'accepted',
        latencyMs: performance.now() - start,
        costCents: 0,
      });
      return fallback;
    }
  }

  private async callModel(prompt: string): Promise<string> {
    // Replace with actual SDK call (OpenAI, Anthropic, Vercel AI, etc.)
    return `AI response for: ${prompt}`;
  }

  private calculateCost(inputTokens: number): number {
    // Simplified cost model
    return Math.ceil(inputTokens * 0.000002 * 100);
  }
}

Architecture Decisions & Rationale

• Decoupled Evaluation Layer: Model benchmarks run independently from product telemetry. This prevents metric contamination and allows model swaps without altering product logic.
• Shadow Mode First: Validates outcome correlation before user exposure. Reduces rollback risk and provides baseline behavioral data.
• Deterministic Fallbacks: Guarantees workflow continuity. Fallback triggers become high-signal PMF indicators; frequent fallbacks indicate misaligned model capability or poor prompt design.
• Cost-Aware Routing: AI PMF fails when operational costs outpace perceived value. Routing lightweight tasks to cheaper models preserves margin and reduces latency.
• Event-Driven Feedback Loop: Behavioral metrics feed directly into model selection criteria. This transforms PMF from a quarterly review into a continuous engineering metric.

Pitfall Guide

1. Optimizing for Accuracy Over Utility Chasing 95%+ accuracy on static benchmarks ignores workflow context. Users tolerate 80% accuracy if the output reduces friction. Best practice: Define a utility threshold based on task complexity, then optimize for latency and cost once the threshold is met.

2. Ignoring Latency & Cost Budgets During Validation PMF validation that excludes latency and cost produces false positives. A model that takes 3 seconds to generate a low-value suggestion will be abandoned. Best practice: Enforce P95 latency budgets in shadow mode and track CPV from day one.

3. Missing Deterministic Fallback Paths Shipping AI without fallbacks guarantees workflow breaks during model degradation or rate limits. Best practice: Every AI route must include a rule-based or cached fallback. Log fallback frequency as a primary PMF signal.

4. Measuring Usage Instead of Value Completion High interaction volume does not equal PMF. Users may click an AI feature out of curiosity, then abandon it. Best practice: Track outcome actions (accept/edit/discard) rather than impressions or API calls.

5. Overcomplicating Prompt Orchestration Multi-agent prompt chains and complex reflection loops increase latency and failure points without proportional utility gains. Best practice: Start with single-turn, constrained prompts. Add complexity only when outcome metrics plateau.

6. Static Evaluation Datasets Relying on fixed test sets causes drift blindness. Production data distribution shifts as user behavior and model versions change. Best practice: Refresh evaluation datasets quarterly using production outcome logs. Automate drift detection via statistical tests on input/output distributions.

7. Treating PMF as a Milestone Instead of a Loop AI PMF degrades as models improve, competitors ship, and user expectations shift. Best practice: Implement continuous evaluation pipelines that trigger model swaps or prompt revisions when CPV or OCR crosses predefined thresholds.

Production Bundle

Action Checklist

• Define outcome metrics: Replace accuracy tracking with outcome completion rate, workflow friction index, and cost-per-value.
• Instrument telemetry layer: Deploy event tracking for trigger context, model output, user action, latency, and cost.
• Deploy shadow mode: Run AI pipeline alongside deterministic logic for 14 days to establish baseline behavioral correlation.
• Implement fallback routing: Configure deterministic or cached fallbacks for every AI route with latency budget enforcement.
• Establish evaluation cadence: Refresh test datasets quarterly using production logs and automate drift detection.
• Set PMF thresholds: Define minimum OCR, maximum WFI, and maximum CPV required to graduate from shadow to production.
• Build feedback loop: Route outcome metrics to model selection and prompt orchestration pipelines for continuous iteration.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High-frequency, low-complexity task	Lightweight model + deterministic fallback	Latency sensitivity outweighs accuracy needs; fallback ensures continuity	Lowers cost by 60-80% per request
Low-frequency, high-stakes task	High-capacity model + human-in-the-loop review	Accuracy and compliance critical; latency budget can be relaxed	Increases cost but reduces liability and churn
New feature validation	Shadow mode + behavioral telemetry	Isolates model risk and validates outcome correlation before user exposure	Neutral initial cost; prevents sunk engineering investment
Degrading PMF metrics	Prompt optimization before model swap	Prompt tuning resolves 70% of outcome drops at 1/10th the cost of retraining	Reduces operational spend while preserving feature
Multi-modal workflow	Route-specific models + unified outcome schema	Different modalities have distinct latency/cost profiles; unified schema enables cross-feature PMF tracking	Increases architectural complexity but improves margin visibility

Configuration Template

# ai-pmf-config.yaml
feature:
  id: "smart-summary"
  version: "1.2.0"
  outcome_schema:
    accepted: "user_applies_summary"
    edited: "user_modifies_summary"
    discarded: "user_closes_without_action"
    ignored: "no_interaction_within_30s"

routing:
  primary:
    provider: "openai"
    model: "gpt-4o-mini"
    latency_budget_ms: 450
    fallback: "rule_based_extractor"
  fallback:
    provider: "local"
    model: "deterministic_extractor"
    latency_budget_ms: 80

telemetry:
  batch_size: 100
  flush_interval_ms: 5000
  endpoint: "https://events.yourdomain.com/ai-outcomes"

evaluation:
  refresh_interval_days: 90
  drift_threshold: 0.15
  metrics:
    - outcome_completion_rate
    - workflow_friction_index
    - cost_per_value
  thresholds:
    min_ocr: 0.65
    max_wfi: 0.35
    max_cpv_cents: 45

Quick Start Guide

1. Initialize the telemetry schema: Copy the AIOutcomeEvent interface and OutcomeTelemetry class into your project. Wire UI interaction hooks to emit outcomeAction events.
2. Configure the router: Instantiate AIRouter with your model SDK, latency budget, and fallback rule. Attach the telemetry instance.
3. Deploy in shadow mode: Route 100% of traffic through the router but display existing UI. Log all model outputs and fallback triggers to your event pipeline.
4. Validate PMF thresholds: After 14 days, calculate OCR, WFI, and CPV. If metrics cross the thresholds in the configuration template, enable user-facing routing. If not, adjust prompts or swap to a lighter model.
5. Activate continuous evaluation: Schedule automated dataset refreshes and drift detection. Route metric violations to your CI/CD pipeline for prompt/model revision triggers.