Fine-tuning vs prompt engineering

ioning, templating, and dynamic few-shot injection.

// prompt-engineering.ts
import { OpenAI } from 'openai';

export class PromptManager {
  private client: OpenAI;
  private version: string;

  constructor(apiKey: string, version: string = 'v1') {
    this.client = new OpenAI({ apiKey });
    this.version = version;
  }

  async executeClassification(input: string, context?: Record<string, any>): Promise<ClassificationResult> {
    // Production prompts should be stored in a versioned store, not hardcoded
    const prompt = this.buildPrompt(input, context);
    
    const response = await this.client.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.1, // Low temp for consistency
      response_format: { type: 'json_object' },
    });

    return JSON.parse(response.choices[0].message.content || '{}');
  }

  private buildPrompt(input: string, context?: Record<string, any>): string {
    const system = `You are a classification engine. Output strictly valid JSON.
    Schema: { category: string, confidence: number, reasoning: string }`;
    
    // Few-shot examples injected dynamically based on input similarity
    const fewShots = this.retrieveFewShots(input);
    
    return `${system}\n\nExamples:\n${fewShots}\n\nInput: ${input}`;
  }

  private retrieveFewShots(input: string): string {
    // In production, use vector search to retrieve top-k similar examples
    return `
      Input: "Server timeout on /api/v1/users"
      Output: {"category": "incident", "confidence": 0.95, "reasoning": "Explicit error mention"}
    `;
  }
}

2. Fine-tuning Implementation

Fine-tuning requires data preparation, job submission, and inference against the fine-tuned model ID.

// fine-tuning.ts
import { OpenAI } from 'openai';
import fs from 'fs';

export class FineTuningOrchestrator {
  private client: OpenAI;

  constructor(apiKey: string) {
    this.client = new OpenAI({ apiKey });
  }

  async prepareAndFineTune(rawData: LabeledExample[]): Promise<string> {
    // 1. Transform to JSONL format required by API
    const jsonlContent = rawData.map(ex => JSON.stringify({
      messages: [
        { role: 'system', content: 'Classify text into categories.' },
        { role: 'user', content: ex.input },
        { role: 'assistant', content: JSON.stringify(ex.output) }
      ]
    })).join('\n');

    const filePath = './training_data.jsonl';
    fs.writeFileSync(filePath, jsonlContent);

    // 2. Upload file
    const file = await this.client.files.create({
      file: fs.createReadStream(filePath),
      purpose: 'fine-tune',
    });

    // 3. Create fine-tuning job
    const job = await this.client.fineTuning.jobs.create({
      model: 'gpt-4o-mini-2024-07-18',
      training_file: file.id,
      hyperparameters: {
        n_epochs: 3,
        batch_size: 4,
        learning_rate_multiplier: 1.0,
      },
    });

    // 4. Poll for completion (simplified)
    const completedJob = await this.pollJob(job.id);
    return completedJob.fine_tuned_model || '';
  }

  async pollJob(jobId: string): Promise<any> {
    // Implementation omitted for brevity; must handle polling with exponential backoff
    return { fine_tuned_model: 'ft:gpt-4o-mini:custom-model-id' };
  }
}

3. Architecture Decision Logic

Integrate decision logic into your inference service to route requests or trigger re-evaluations.

// decision-engine.ts
export interface ModelMetrics {
  monthlyTokens: number;
  avgLatencyMs: number;
  formatErrorRate: number;
  devCostHours: number;
}

export function recommendStrategy(metrics: ModelMetrics): 'prompt' | 'fine-tune' | 'hybrid' {
  // Thresholds derived from production analysis
  const TOKEN_CUTOFF = 5_000_000; // 5M tokens/month
  const LATENCY_SLA = 600; // ms
  const FORMAT_ERROR_THRESHOLD = 0.05; // 5%

  const costSavings = (metrics.monthlyTokens / 1_000_000) * (14.50 - 5.20);
  const devCost = metrics.devCostHours * 150; // Assume $150/hr engineering cost

  // ROI Check: Fine-tuning pays off if cost savings exceed dev cost within 3 months
  const roiMonths = devCost / (costSavings / 3);

  if (metrics.avgLatencyMs > LATENCY_SLA || metrics.formatErrorRate > FORMAT_ERROR_THRESHOLD) {
    if (roiMonths <= 3) {
      return 'fine-tune';
    }
    return 'hybrid'; // FT for format, PE for dynamic constraints
  }

  if (metrics.monthlyTokens < TOKEN_CUTOFF) {
    return 'prompt';
  }

  return 'hybrid';
}

Pitfall Guide

1. Fine-tuning for Knowledge Injection:

   • Mistake: Using fine-tuning to teach the model new facts, policies, or proprietary data.
   • Impact: The model will hallucinate, forget the knowledge quickly, or fail to retrieve it reliably. Fine-tuning alters weights for patterns, not retrieval.
   • Best Practice: Use RAG (Retrieval-Augmented Generation) for knowledge. Fine-tune only for style, tone, format, and task-specific reasoning patterns.

2. Ignoring Prompt Versioning and Drift:

   • Mistake: Treating prompts as static strings in code without version control or A/B testing infrastructure.
   • Impact: Model updates can break prompts. Without versioning, you cannot rollback or measure the impact of prompt changes.
   • Best Practice: Store prompts in a dedicated prompt management system with version hashes. Implement automated evaluation suites that run against prompt changes before deployment.

3. Catastrophic Forgetting in Fine-tuning:

   • Mistake: Fine-tuning on a narrow dataset with too many epochs or high learning rates.
   • Impact: The model loses general capabilities, becoming brittle on out-of-distribution inputs.
   • Best Practice: Use conservative hyperparameters. Validate against a hold-out set that includes general tasks. Consider LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning to preserve base capabilities.

4. Data Leakage in Evaluation:

   • Mistake: Including test examples in the training set or using similar distributions for evaluation.
   • Impact: Inflated accuracy metrics leading to poor production performance.
   • Best Practice: Strictly partition data into train/validation/test sets. Ensure test data represents edge cases and distribution shifts expected in production. Use cross-validation for small datasets.

5. Underestimating Data Engineering Costs:

   • Mistake: Assuming fine-tuning is "easy" because APIs abstract the training loop.
   • Impact: Projects stall due to poor data quality. LLMs are garbage-in-garbage-out machines.
   • Best Practice: Allocate 60% of fine-tuning effort to data curation. Clean, format, and augment data. Generate synthetic data using stronger models to bootstrap training sets if real data is scarce.

6. Over-Optimizing for Accuracy vs. Latency/Cost:

   • Mistake: Chasing marginal accuracy gains with complex prompts or larger models when latency constraints are violated.
   • Impact: Poor user experience and runaway cloud bills.
   • Best Practice: Define SLAs for latency and cost first. Optimize for the smallest model and simplest approach that meets SLAs. Use fine-tuning to enable smaller models to perform at the level of larger models.

7. Catastrophic Prompt Bloat:

   • Mistake: Adding endless constraints, examples, and instructions to a prompt to fix edge cases.
   • Impact: Exponential cost increase, attention dilution, and model confusion.
   • Best Practice: If a prompt exceeds 2,000 tokens or requires more than 5 few-shot examples, evaluate fine-tuning. Compress prompts by removing redundant instructions and using structured formats.

Production Bundle

Action Checklist

• Audit Token Usage: Analyze current prompt token distribution and identify bloat sources.
• Benchmark Baseline: Run automated evaluations on prompt engineering accuracy, latency, and cost.
• Define SLAs: Establish hard limits for latency, cost per request, and format adherence.
• Evaluate Data Readiness: Assess volume, quality, and labeling consistency for potential fine-tuning datasets.
• Calculate ROI: Use the decision matrix to compare TCO of prompt engineering vs. fine-tuning over 6 months.
• Implement Hybrid Strategy: If fine-tuning, retain prompts for dynamic constraints and RAG context.
• Set Up Monitoring: Deploy drift detection and automated evaluation pipelines for the chosen approach.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Low Volume (<1M tokens/mo)	Prompt Engineering	Dev overhead of FT outweighs token savings.	Low
High Volume (>10M tokens/mo)	Fine-tuning	Token savings offset dev costs; latency improves significantly.	High Savings
Strict Format (JSON/XML)	Fine-tuning	FT internalizes structure, reducing prompt verbosity and errors.	Medium Savings
Dynamic Knowledge Updates	Prompting + RAG	FT cannot handle real-time updates; RAG provides fresh context.	Medium
Latency Sensitive (<500ms)	Fine-tuning	Shorter prompts reduce inference time; smaller models suffice.	High Savings
Complex Reasoning Tasks	Prompt Engineering + Chain-of-Thought	FT improves format, not reasoning depth; CoT prompts guide logic.	Low
Style/Tone Consistency	Fine-tuning	FT aligns output distribution to desired voice better than prompts.	Medium

Configuration Template

Ready-to-use configuration for a hybrid inference service that routes based on strategy.

// config/inference-config.ts
export interface InferenceConfig {
  strategy: 'prompt' | 'fine-tune' | 'hybrid';
  model: string;
  promptTemplateId?: string;
  maxTokens?: number;
  temperature?: number;
  responseFormat?: 'text' | 'json';
  ragEnabled?: boolean;
  ragCollection?: string;
}

export const defaultConfigs: Record<string, InferenceConfig> = {
  'classification-task': {
    strategy: 'fine-tune',
    model: 'ft:gpt-4o-mini:classification:v1',
    maxTokens: 100,
    temperature: 0.1,
    responseFormat: 'json',
  },
  'support-assistant': {
    strategy: 'hybrid',
    model: 'ft:gpt-4o-mini:assistant:v1',
    promptTemplateId: 'support-system-prompt-v3',
    maxTokens: 500,
    temperature: 0.3,
    ragEnabled: true,
    ragCollection: 'knowledge-base',
  },
  'ad-hoc-analysis': {
    strategy: 'prompt',
    model: 'gpt-4o',
    promptTemplateId: 'analysis-template',
    maxTokens: 2000,
    temperature: 0.7,
  },
};

Quick Start Guide

1. Define Task and Metrics: Specify the input/output schema, latency SLA, and target accuracy. Create a test dataset with 50-100 representative examples.
2. Benchmark Prompt Engineering: Implement the task using prompt engineering. Run the test dataset and record latency, cost, and accuracy.
3. Calculate Thresholds: Estimate monthly volume. If volume × (PE Cost - FT Cost) > Dev Cost, proceed to fine-tuning evaluation.
4. Prepare Fine-tuning Data: If proceeding, curate 50-100 high-quality examples. Format as JSONL. Submit a fine-tuning job.
5. Evaluate and Deploy: Compare the fine-tuned model against the prompt baseline. If metrics improve and ROI is positive, deploy the fine-tuned model with a minimal prompt for dynamic constraints. Monitor performance for 2 weeks.