config/aivr-config.yaml

old score are blocked from development or require executive exception. 5. Continuous Recalculation: As model performance degrades or inference costs fluctuate, the engine should trigger re-evaluation of active features.

Code Example: AIVR Scoring Engine

The following TypeScript implementation provides a robust scoring engine. It uses a weighted scoring algorithm with validation gates.

// src/ai-prioritization/AIVRScorer.ts

export interface AIVRFeature {
  id: string;
  name: string;
  // Traditional metrics
  businessValue: number; // 1-10
  userReach: number;     // 1-10
  // AI-specific metrics
  dataReadinessScore: number; // 0-1 (based on data availability, quality, labels)
  inferenceLatencyMs: number;
  inferenceCostPerCall: number; // USD
  modelReliabilityScore: number; // 0-1 (accuracy/f1/hallucination rate)
  operationalComplexity: number; // 1-5 (1=low, 5=high)
  riskCategory: 'LOW' | 'MEDIUM' | 'HIGH';
}

export interface AIVRConfig {
  weights: {
    businessValue: number;
    userReach: number;
    dataReadiness: number;
    inferenceCost: number;
    modelReliability: number;
    operationalComplexity: number;
    risk: number;
  };
  thresholds: {
    minScore: number;
    maxLatencyMs: number;
    maxCostPerCall: number;
    blockedRiskCategories: string[];
  };
}

export class AIVRScorer {
  constructor(private config: AIVRConfig) {}

  public calculate(feature: AIVRFeature): { score: number; grade: string; flags: string[] } {
    const flags: string[] = [];
    
    // 1. Validation Gates (Hard blocks)
    if (this.config.thresholds.blockedRiskCategories.includes(feature.riskCategory)) {
      return { score: 0, grade: 'F', flags: ['BLOCKED: High Risk Category'] };
    }
    if (feature.inferenceLatencyMs > this.config.thresholds.maxLatencyMs) {
      flags.push('WARNING: Latency exceeds SLA');
    }
    if (feature.inferenceCostPerCall > this.config.thresholds.maxCostPerCall) {
      flags.push('WARNING: Cost exceeds budget');
    }

    // 2. Normalized Scoring
    const valueScore = (
      (feature.businessValue * this.config.weights.businessValue) +
      (feature.userReach * this.config.weights.userReach)
    ) / (this.config.weights.businessValue + this.config.weights.userReach);

    const feasibilityScore = (
      (feature.dataReadinessScore * this.config.weights.dataReadiness) +
      ((6 - feature.operationalComplexity) / 5 * this.config.weights.operationalComplexity)
    ) / (this.config.weights.dataReadiness + this.config.weights.operationalComplexity);

    // Cost efficiency: Lower cost is better. Invert score.
    const costScore = Math.max(0, 1 - (feature.inferenceCostPerCall / 10)) * this.config.weights.inferenceCost;
    
    // Reliability: Higher is better
    const reliabilityScore = feature.modelReliabilityScore * this.config.weights.modelReliability;

    // 3. Composite Score
    const totalWeight = this.config.weights.inferenceCost + this.config.weights.modelReliability;
    const riskAdjustedScore = (costScore + reliabilityScore) / totalWeight;

    const finalScore = (valueScore * 0.4) + (feasibilityScore * 0.3) + (riskAdjustedScore * 0.3);

    // 4. Grading
    let grade = 'F';
    if (finalScore >= 0.8) grade = 'A';
    else if (finalScore >= 0.6) grade = 'B';
    else if (finalScore >= 0.4) grade = 'C';
    else if (finalScore >= 0.2) grade = 'D';

    return {
      score: finalScore,
      grade,
      flags: flags.concat(finalScore < this.config.thresholds.minScore ? ['BLOCKED: Below minimum score'] : [])
    };
  }
}

Architecture Decisions and Rationale

• Policy-as-Code: The scoring logic is implemented in code rather than static spreadsheets. This allows versioning, automated testing, and integration with infrastructure. It ensures that prioritization criteria are treated with the same rigor as application code.
• Decoupled Metrics: The AIVRFeature interface separates business metrics from AI metrics. This allows product managers to update value scores independently while data engineers update readiness scores. The engine aggregates these inputs without tight coupling.
• Hard Gates vs. Soft Weights: The architecture distinguishes between hard gates (e.g., risk category, latency SLA) and weighted scores. Hard gates prevent features that are fundamentally unshippable from consuming evaluation cycles. Soft weights allow trade-offs between value and cost where appropriate.
• Extensibility: The configuration object allows teams to adjust weights based on strategic shifts. For example, a cost-optimization phase can increase the weight of inferenceCost, automatically reprioritizing the backlog without manual re-evaluation of every ticket.

Pitfall Guide

1. Ignoring Data Labeling Costs:

   • Mistake: Prioritizing features based on raw data availability without assessing label quality or cost.
   • Impact: Development stalls when the team discovers that 80% of the data requires manual annotation, inflating effort estimates by 300%.
   • Practice: Include "Label Readiness" as a mandatory field in the feature spec. Estimate labeling costs using active learning simulations.

2. Optimizing for Accuracy Over Latency:

   • Mistake: Selecting the largest model because it has the highest benchmark score.
   • Impact: Inference latency exceeds user tolerance, leading to abandonment. The feature is technically accurate but product-failed.
   • Practice: Define latency budgets during prioritization. Evaluate small models or quantization strategies early. Accuracy gains must justify latency penalties.

3. Treating AI as Binary:

   • Mistake: Assuming an AI feature is either "working" or "broken."
   • Impact: Teams build brittle features that degrade gracefully only in ideal conditions.
   • Practice: Prioritize features that include fallback mechanisms, confidence thresholds, and progressive enhancement. Score features higher if they define graceful degradation paths.

4. Neglecting Model Drift in ROI Calculations:

   • Mistake: Calculating ROI based on initial model performance without accounting for drift.
   • Impact: Features become less valuable over time as data distributions shift, turning positive ROI features into cost centers.
   • Practice: Factor in retraining frequency and monitoring costs. Prioritize features with stable data distributions or automated drift detection.

5. User Trust Mismatch:

   • Mistake: Marketing AI capabilities that exceed the model's actual reliability.
   • Impact: High user churn when the model hallucinates or fails on edge cases. Trust recovery is expensive.
   • Practice: Align prioritization with "Trust Metrics." Features with high hallucination risk must include guardrails or disclaimers. Do not prioritize high-visibility features with low reliability scores.

6. Vendor Lock-in Blindness:

   • Mistake: Prioritizing features that tightly couple the product to a specific proprietary model API.
   • Impact: Inability to switch vendors due to cost increases or API changes. Strategic vulnerability.
   • Practice: Score features based on abstraction layer maturity. Prioritize features that use model-agnostic interfaces or open-weight models where feasible.

7. Forgetting the "Human-in-the-Loop" Cost:

   • Mistake: Assuming full automation.
   • Impact: Operational teams are overwhelmed by error correction workflows.
   • Practice: Estimate the volume of exceptions and the cost of human review. Prioritize features where the exception rate is manageable or the cost of review is low.

Production Bundle

Action Checklist

• Audit Data Readiness: Verify that training and inference data pipelines are established, versioned, and meet quality standards before committing to development.
• Define SLA Constraints: Establish hard limits for latency, availability, and cost per inference. Integrate these into the prioritization thresholds.
• Calculate TCO: Estimate total cost of ownership including inference, storage, guardrails, monitoring, and retraining. Compare TCO against projected revenue.
• Assess Risk Profile: Classify features by risk category (Low/Medium/High) based on safety, compliance, and brand impact. Block High risk features without mitigation plans.
• Implement Fallback Strategy: Ensure every prioritized feature has a defined fallback mechanism for model failure or low confidence.
• Configure Scoring Engine: Deploy the AIVR scorer with weights and thresholds aligned to current business objectives.
• Review Backlog: Run the scoring engine against the current backlog. Reprioritize based on AIVR scores, not just business value.
• Establish Drift Monitoring: Set up automated monitoring for model performance metrics. Trigger re-prioritization if drift exceeds thresholds.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
High Value, Low Risk, High Data Readiness	Immediate Development	Low friction, high ROI. Ideal candidate for rapid shipping.	Low marginal cost; high return.
High Value, High Risk, Low Data Readiness	Research Spike / Pilot	Requires significant investment in data and risk mitigation. Do not commit to full build.	High initial investment; potential for high cost if risks materialize.
Low Value, Low Risk, High Data Readiness	Defer or Automate	Easy to build but low impact. Automate if possible or defer to fill capacity gaps.	Low cost, but low efficiency of engineering resources.
High Value, Medium Risk, Medium Data Readiness	Phased Rollout	Build MVP with guardrails. Validate risk assumptions before scaling.	Moderate cost; mitigates risk through incremental investment.
Low Value, High Risk, Any Data Readiness	Reject	Negative ROI and potential brand damage. No justification for resource allocation.	Zero cost; avoids waste and liability.

Configuration Template

Copy this configuration to initialize the AIVR Scorer for a standard SaaS product line. Adjust weights based on strategic focus.

# config/aivr-config.yaml
ai_prioritization:
  weights:
    business_value: 0.25
    user_reach: 0.15
    data_readiness: 0.20
    inference_cost: 0.15
    model_reliability: 0.15
    operational_complexity: 0.10
  
  thresholds:
    min_score: 0.45
    max_latency_ms: 500
    max_cost_per_call: 0.02
    blocked_risk_categories:
      - HIGH
  
  data_requirements:
    min_label_coverage: 0.8
    max_data_age_days: 30
    required_fields:
      - text_content
      - user_context
  
  risk_policies:
    high_risk_features:
      require_guardrails: true
      require_human_review: true
      max_hallucination_rate: 0.01

Quick Start Guide

1. Install Dependencies:

   npm install @codcompass/ai-prioritization
   

2. Initialize Configuration: Create aivr.config.ts using the Configuration Template. Adjust weights and thresholds to match your product requirements.

3. Integrate Scorer: Import the AIVRScorer and run against your feature backlog.

   import { AIVRScorer } from '@codcompass/ai-prioritization';
   import config from './aivr.config';
   
   const scorer = new AIVRScorer(config);
   const results = backlog.map(feature => scorer.calculate(feature));
   
   // Sort by score descending
   results.sort((a, b) => b.score - a.score);
   

4. Enforce in CI/CD: Add a step to your pipeline that fails if a new AI feature PR does not include an AIVR score above the threshold.

   # .github/workflows/ai-prioritization.yml
   steps:
     - name: Check AI Feature Score
       run: |
         SCORE=$(node scripts/check-score.js $PR_FEATURE_ID)
         if [ "$SCORE" -lt "$MIN_SCORE" ]; then
           echo "AI Feature score $SCORE is below threshold $MIN_SCORE"
           exit 1
         fi
   

5. Dashboard Integration: Export scores to your project management tool via API to visualize prioritization changes and track AI health metrics over time.