ai-success-metrics-config.yaml

Use automated LLM-as-judge scoring for high-volume interactions, paired with human-in-the-loop validation for edge cases. Automated scoring handles consistency; human validation calibrates thresholds and catches systemic drift.

3. Build metric aggregation pipeline Stream events to a processing layer that computes rolling metrics, segments by user persona, and applies dynamic thresholds. Store results in a query-optimized datastore for dashboarding and alerting.

4. Configure feedback loops Route low-success signals back to product and engineering queues. Trigger re-evaluation when drift is detected, not when arbitrary time windows expire.

Code Examples (TypeScript)

Event Ingestion Schema

interface AIInteractionEvent {
  traceId: string;
  userId: string;
  sessionId: string;
  featureId: string;
  timestamp: ISOString;
  
  // Intent & context
  taskType: 'summarization' | 'extraction' | 'generation' | 'classification';
  userPrompt: string;
  modelId: string;
  modelOutput: string;
  
  // Outcome signals
  accepted: boolean;
  edited: boolean;
  editDelta: string | null;
  fallbackTriggered: boolean;
  fallbackReason: string | null;
  latencyMs: number;
  tokenCount: number;
}

type ISOString = string;

Automated Evaluation Scorer

import { createClient } from '@supabase/supabase-js';

const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_KEY!);

interface EvaluationScore {
  traceId: string;
  taskCompletion: number; // 0-1
  relevance: number;      // 0-1
  safety: number;         // 0-1
  overall: number;        // weighted average
}

export async function scoreInteraction(event: AIInteractionEvent): Promise<EvaluationScore> {
  // LLM-as-judge evaluation prompt
  const evaluationPrompt = `
    Evaluate the AI response against the user intent.
    Task: ${event.taskType}
    User Prompt: ${event.userPrompt}
    Model Output: ${event.modelOutput}
    
    Return JSON with scores (0-1) for:
    - taskCompletion: Did the output achieve the stated task?
    - relevance: Is the output directly applicable to the prompt?
    - safety: Does the output avoid hallucination, bias, or harmful content?
  `;

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: { 'Authorization': `Bearer ${process.env.OPENAI_API_KEY}` },
    body: JSON.stringify({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: evaluationPrompt }],
      response_format: { type: 'json_object' }
    })
  });

  const data = await response.json();
  const scores = JSON.parse(data.choices[0].message.content);

  const overall = (scores.taskCompletion * 0.5) + (scores.relevance * 0.3) + (scores.safety * 0.2);

  return {
    traceId: event.traceId,
    taskCompletion: scores.taskCompletion,
    relevance: scores.relevance,
    safety: scores.safety,
    overall: Math.round(overall * 100) / 100
  };
}

Metric Aggregation Service

export class AIMetricAggregator {
  private readonly windowMs = 3600000; // 1 hour rolling window

  async computeSuccessRate(events: AIInteractionEvent[]): Promise<number> {
    const valid = events.filter(e => !e.fallbackTriggered);
    const successful = valid.filter(e => e.accepted || (!e.accepted && !e.edited));
    return valid.length > 0 ? successful.length / valid.length : 0;
  }

  async computeSilentFailureRate(events: AIInteractionEvent[]): Promise<number> {
    const rejected = events.filter(e => e.accepted === false && e.fallbackTriggered === false);
    return events.length > 0 ? rejected.length / events.length : 0;
  }

  async storeMetrics(traceId: string, scores: EvaluationScore, successRate: number, failureRate: number): Promise<void> {
    await supabase.from('ai_success_metrics').insert({
      trace_id: traceId,
      overall_score: scores.overall,
      task_completion: scores.taskCompletion,
      relevance: scores.relevance,
      safety: scores.safety,
      success_rate: successRate,
      silent_failure_rate: failureRate,
      recorded_at: new Date().toISOString()
    });
  }
}

Architecture Decisions and Rationale

• Decoupled evaluation pipeline: Real-time inference must remain under 2 seconds. Evaluation runs asynchronously via message queue (Kafka, SQS, or Pub/Sub). This prevents scoring overhead from degrading user experience.
• Weighted scoring over binary pass/fail: AI outputs exist on a spectrum. Weighted aggregation (task completion 50%, relevance 30%, safety 20%) aligns with product priorities. Safety carries lower weight by default because most production models enforce baseline guardrails; task completion drives retention.
• Rolling windows with persona segmentation: Aggregate metrics per user tier, feature, and workflow. A 60% success rate for power users may indicate a bug, while the same rate for new users may indicate onboarding friction. Static thresholds create false alarms.
• Feedback-driven threshold calibration: Thresholds adjust based on acceptance patterns, not arbitrary benchmarks. If edit rate drops below 8% for a feature, the success threshold can tighten. If fallback spikes, the system triggers re-evaluation of the prompt template or model routing.

Pitfall Guide

1. Tracking latency and cost as primary success indicators Latency and token cost measure operational efficiency, not user value. Optimizing for sub-500ms response times while ignoring task completion produces fast, useless outputs. Best practice: Tie cost and latency to success rates. Report cost-per-successful-interaction, not cost-per-call.

2. Ignoring fallback and escalation rates When users hit fallback triggers (human handoff, rule-based path, or manual override), the AI failed the task. Teams that log fallbacks as "edge cases" miss systemic routing failures. Best practice: Treat fallback rate as a leading indicator of churn. Route fallback reasons to prompt engineering queues.

3. Conflating model accuracy with user task success A model can achieve 94% accuracy on a benchmark dataset while failing in production because the benchmark doesn't match user intent. Production success depends on workflow integration, not isolated accuracy. Best practice: Evaluate against actual user prompts, not synthetic test sets. Segment by task type and user role.

4. Hardcoding static thresholds without persona segmentation A 70% success threshold may be acceptable for internal tools but catastrophic for customer-facing features. Static thresholds ignore usage patterns and tolerance levels. Best practice: Implement dynamic thresholds based on historical acceptance rates per segment. Alert on deviation, not absolute values.

5. Missing the cost-to-value ratio per interaction Teams track total AI spend but rarely map it to resolved tickets, generated revenue, or time saved. Without cost-to-value mapping, scaling decisions become guesswork. Best practice: Tag each interaction with business outcome proxies (ticket closed, document generated, decision made). Calculate ROI per feature, not per model.

6. Not instrumenting explicit feedback loops Implicit signals (acceptance, edits, time-on-page) are necessary but insufficient. Users rarely correct AI outputs explicitly unless prompted. Best practice: Implement lightweight feedback mechanisms (thumbs up/down, quick edit capture, intent mismatch flag). Route negative feedback to prompt revision pipelines within 24 hours.

7. Treating evaluation as post-deployment only Evaluation shouldn't start after launch. Pre-deployment simulation, shadow mode testing, and canary routing catch alignment gaps before users experience them. Best practice: Run evaluation in shadow mode for 7-14 days. Compare shadow scores against production scores. Only promote when delta falls below 5%.

Production Bundle

Action Checklist

• Define outcome-based event schema: Capture task type, prompt, output, acceptance, edits, and fallback signals in every AI interaction.
• Implement async evaluation pipeline: Route events to a message queue. Run LLM-as-judge scoring and human validation asynchronously to preserve inference latency.
• Configure weighted scoring model: Assign weights to task completion, relevance, and safety based on product priorities. Store scores in a query-optimized datastore.
• Segment metrics by persona and feature: Compute success rates per user tier, workflow, and model version. Avoid aggregate-only dashboards.
• Instrument explicit feedback mechanisms: Add thumbs up/down, edit capture, and intent mismatch flags. Route negative signals to prompt engineering queues.
• Establish dynamic thresholds: Replace static pass/fail benchmarks with rolling acceptance rates. Alert on deviation, not absolute values.
• Run shadow mode validation: Deploy evaluation in shadow mode for 7-14 days. Compare shadow scores against production before full rollout.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Low-volume, high-stakes feature (e.g., legal drafting)	Human-in-the-loop evaluation with strict safety thresholds	Manual review catches nuanced failures that LLM judges miss. Safety outweighs speed.	High per-interaction cost, low churn risk
High-volume, transactional feature (e.g., ticket routing)	Automated LLM-as-judge scoring with rolling acceptance thresholds	Scale requires automation. Dynamic thresholds adapt to usage patterns without manual overhead.	Low per-interaction cost, requires queue infrastructure
Multi-model routing environment	Feature-level success scoring with model version tagging	Isolates which model drives success per workflow. Prevents blanket model upgrades that degrade specific tasks.	Moderate cost for tracking and A/B routing
Early-stage AI feature (<10k interactions)	Shadow mode + explicit feedback collection	Validates alignment before user exposure. Feedback loops calibrate thresholds without production risk.	Low initial cost, delays launch by 7-14 days

Configuration Template

# ai-success-metrics-config.yaml
metrics:
  scoring:
    weights:
      task_completion: 0.5
      relevance: 0.3
      safety: 0.2
    evaluation_model: "gpt-4o-mini"
    max_concurrent_jobs: 50
    timeout_seconds: 15

  thresholds:
    mode: "dynamic" # static | dynamic
    baseline_window_hours: 168
    alert_deviation_percent: 12
    fallback_alert_threshold: 0.08

  segmentation:
    dimensions:
      - "user_tier"
      - "feature_id"
      - "model_version"
      - "task_type"
    retention_cooldown_hours: 72

pipeline:
  ingestion:
    queue: "ai-interactions"
    format: "json"
    schema_version: "v2"
  evaluation:
    async_worker: true
    human_review_sample_rate: 0.05
    retry_attempts: 3
  storage:
    table: "ai_success_metrics"
    retention_days: 90
    aggregation_interval: "1h"

Quick Start Guide

1. Install the event SDK: Add the TypeScript tracking package to your frontend and backend. Initialize with your project ID and enable AI interaction capture.

   npm install @codcompass/ai-metrics-sdk
   

   import { AIMetricsTracker } from '@codcompass/ai-metrics-sdk';
   const tracker = new AIMetricsTracker({ projectId: 'your-project-id' });
   

2. Wire the event emitter: Call trackInteraction after every AI response. Pass prompt, output, acceptance state, and fallback flags.

   tracker.trackInteraction({
     traceId: crypto.randomUUID(),
     userId: session.user.id,
     taskType: 'extraction',
     userPrompt: prompt,
     modelOutput: response,
     accepted: userAccepted,
     fallbackTriggered: fallbackUsed,
     latencyMs: Date.now() - startTime
   });
   

3. Deploy the evaluation worker: Run the async scoring service using the provided configuration template. Point it to your message queue and evaluation model API key.

   docker run --env-file .env codcompass/ai-eval-worker:latest
   

4. Verify metric ingestion: Check the dashboard for rolling success rates, silent failure rates, and segment breakdowns. Confirm that acceptance signals map correctly to your product workflows.

5. Activate alerting: Configure webhook or Slack notifications for threshold deviations. Route fallback spikes to your prompt engineering backlog. Review weekly calibration reports to adjust weights and thresholds.