Fine-tuning vs prompt engineering

   { role: 'user', content: ex.input },
    { role: 'assistant', content: JSON.stringify(ex.output) }
  ]
})).join('\n');

const filePath = './training_data.jsonl';
fs.writeFileSync(filePath, jsonlContent);

// 2. Upload file
const file = await this.client.files.create({
  file: fs.createReadStream(filePath),
  purpose: 'fine-tune',
});

// 3. Create fine-tuning job
const job = await this.client.fineTuning.jobs.create({
  model: 'gpt-4o-mini-2024-07-18',
  training_file: file.id,
  hyperparameters: {
    n_epochs: 3,
    batch_size: 4,
    learning_rate_multiplier: 1.0,
  },
});

// 4. Poll for completion (simplified)
const completedJob = await this.pollJob(job.id);
return completedJob.fine_tuned_model || '';

}

async pollJob(jobId: string): Promise<any> { // Implementation omitted for brevity; must handle polling with exponential backoff return { fine_tuned_model: 'ft:gpt-4o-mini:custom-model-id' }; } }

### 3. Architecture Decision Logic
Integrate decision logic into your inference service to route requests or trigger re-evaluations.

```typescript
// decision-engine.ts
export interface ModelMetrics {
  monthlyTokens: number;
  avgLatencyMs: number;
  formatErrorRate: number;
  devCostHours: number;
}

export function recommendStrategy(metrics: ModelMetrics): 'prompt' | 'fine-tune' | 'hybrid' {
  // Thresholds derived from production analysis
  const TOKEN_CUTOFF = 5_000_000; // 5M tokens/month
  const LATENCY_SLA = 600; // ms
  const FORMAT_ERROR_THRESHOLD = 0.05; // 5%

  const costSavings = (metrics.monthlyTokens / 1_000_000) * (14.50 - 5.20);
  const devCost = metrics.devCostHours * 150; // Assume $150/hr engineering cost

  // ROI Check: Fine-tuning pays off if cost savings exceed dev cost within 3 months
  const roiMonths = devCost / (costSavings / 3);

  if (metrics.avgLatencyMs > LATENCY_SLA || metrics.formatErrorRate > FORMAT_ERROR_THRESHOLD) {
    if (roiMonths <= 3) {
      return 'fine-tune';
    }
    return 'hybrid'; // FT for format, PE for dynamic constraints
  }

  if (metrics.monthlyTokens < TOKEN_CUTOFF) {
    return 'prompt';
  }

  return 'hybrid';
}

Pitfall Guide

1. Fine-tuning for Knowledge Injection:

   • Mistake: Using fine-tuning to teach the model new facts, policies, or proprietary data.
   • Impact: The model will hallucinate, forget the knowledge quickly, or fail to retrieve it reliably. Fine-tuning alters weights for patterns, not retrieval.
   • Best Practice: Use RAG (Retrieval-Augmented Generation) for knowledge. Fine-tune only for style, tone, format, and task-specific reasoning patterns.

2. Ignoring Prompt Versioning and Drift:

   • Mistake: Treating prompts as static strings in code without version control or A/B testing infrastructure.
   • Impact: Model updates can break prompts. Without versioning, you cannot rollback or measure the impact of prompt changes.
   • Best Practice: Store prompts in a dedicated prompt management system with version hashes. Implement automated evaluation suites that run against prompt changes before deployment.

3. Catastrophic Forgetting in Fine-tuning:

   • Mistake: Fine-tuning on a narrow dataset with too many epochs or high learning rates.
   • Impact: The model loses general capabilities, becoming brittle on out-of-distribution inputs.
   • Best Practice: Use conservative hyperparameters. Validate against a hold-out set that includes general tasks. Consider LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning to preserve base capabilities.

4. Data Leakage in Evaluation:

   • Mistake: Including test examples in the training set or using similar distributions for evaluation.
   • Impact: Inflated accuracy metrics leading to poor production performance.
   • Best Practice: Strictly partition data into train/validation/test sets. Ensure test data represents edge cases and distribution shifts expected in production. Use cross-validation for small datasets.

5. Underestimating Data Engineering Costs:

   • Mistake: Assuming fine-tuning is "easy" because APIs abstract the training loop.
   • Impact: Projects stall due to poor data quality. LLMs are garbage-in-garbage-out machines.
   • Best Practice: Allocate 60% of fine-tuning effort to data curation. Clean, format, and augment data. Generate synthetic data using stronger models to bootstrap training sets if real data is scarce.

6. Over-Optimizing for Accuracy vs. Latency/Cost:

   • Mistake: Chasing marginal accuracy gains with complex prompts or larger models when latency constraints are violated.
   • Impact: Poor user experience and runaway cloud bills.
   • Best Practice: Define SLAs for latency and cost first. Optimize for the smallest model and simplest approach that meets SLAs. Use fine-tuning to enable smaller models to perform at the level of larger models.

7. Catastrophic Prompt Bloat:

   • Mistake: Adding endless constraints, examples, and instructions to a prompt to fix edge cases.
   • Impact: Exponential cost increase, attention dilution, and model confusion.
   • Best Practice: If a prompt exceeds 2,000 tokens or requires more than 5 few-shot examples, evaluate fine-tuning. Compress prompts by removing redundant instructions and using structured formats.

Production Bundle

Action Checklist

• Audit Token Usage: Analyze current prompt token distribution and identify bloat sources.
• Benchmark Baseline: Run automated evaluations on prompt engineering accuracy, latency, and cost.
• Define SLAs: Establish hard limits for latency, cost per request, and format adherence.
• Evaluate Data Readiness: Assess volume, quality, and labeling consistency for potential fine-tuning datasets.
• Calculate ROI: Use the decision matrix to compare TCO of prompt engineering vs. fine-tuning over 6 months.
• Implement Hybrid Strategy: If fine-tuning, retain prompts for dynamic constraints and RAG context.
• Set Up Monitoring: Deploy drift detection and automated evaluation pipelines for the chosen approach.

Decision Matrix

Configuration Template

Ready-to-use configuration for a hybrid inference service that routes based on strategy.

// config/inference-config.ts
export interface InferenceConfig {
  strategy: 'prompt' | 'fine-tune' | 'hybrid';
  model: string;
  promptTemplateId?: string;
  maxTokens?: number;
  temperature?: number;
  responseFormat?: 'text' | 'json';
  ragEnabled?: boolean;
  ragCollection?: string;
}

export const defaultConfigs: Record<string, InferenceConfig> = {
  'classification-task': {
    strategy: 'fine-tune',
    model: 'ft:gpt-4o-mini:classification:v1',
    maxTokens: 100,
    temperature: 0.1,
    responseFormat: 'json',
  },
  'support-assistant': {
    strategy: 'hybrid',
    model: 'ft:gpt-4o-mini:assistant:v1',
    promptTemplateId: 'support-system-prompt-v3',
    maxTokens: 500,
    temperature: 0.3,
    ragEnabled: true,
    ragCollection: 'knowledge-base',
  },
  'ad-hoc-analysis': {
    strategy: 'prompt',
    model: 'gpt-4o',
    promptTemplateId: 'analysis-template',
    maxTokens: 2000,
    temperature: 0.7,
  },
};

Quick Start Guide

1. Define Task and Metrics: Specify the input/output schema, latency SLA, and target accuracy. Create a test dataset with 50-100 representative examples.
2. Benchmark Prompt Engineering: Implement the task using prompt engineering. Run the test dataset and record latency, cost, and accuracy.
3. Calculate Thresholds: Estimate monthly volume. If volume × (PE Cost - FT Cost) > Dev Cost, proceed to fine-tuning evaluation.
4. Prepare Fine-tuning Data: If proceeding, curate 50-100 high-quality examples. Format as JSONL. Submit a fine-tuning job.
5. Evaluate and Deploy: Compare the fine-tuned model against the prompt baseline. If metrics improve and ROI is positive, deploy the fine-tuned model with a minimal prompt for dynamic constraints. Monitor performance for 2 weeks.