Embedding model selection guide

ks deploying a model that is expensive, slow, and inaccurate for your specific data distribution.

Core Solution

Implementing a robust embedding selection strategy requires a systematic evaluation pipeline and an abstraction layer that allows model swapping without architectural refactoring.

Step 1: Define Evaluation Metrics

Stop relying on aggregate benchmarks. Create a domain-specific evaluation set containing:

• Queries: Real user queries or synthetic queries generated from your knowledge base.
• Ground Truth: Manually verified relevant document IDs for each query.
• Metrics: Calculate Recall@K, Mean Reciprocal Rank (MRR), and NDCG@K.

Step 2: Benchmark Candidate Models

Run candidates against your evaluation set. Focus on models that fit your infrastructure constraints (e.g., local execution for privacy).

// benchmark-runner.ts
import { createClient } from '@local/embedding-client';
import { cosineSimilarity } from './math-utils';

interface EvaluationResult {
  modelId: string;
  recallAtK: number;
  mrr: number;
  avgLatencyMs: number;
}

export async function runBenchmark(
  candidates: string[],
  evalSet: { query: string; relevantDocs: string[] }[],
  k: number = 5
): Promise<EvaluationResult[]> {
  const results: EvaluationResult[] = [];

  for (const modelId of candidates) {
    const client = createClient({ model: modelId });
    let totalRecall = 0;
    let totalMRR = 0;
    let totalLatency = 0;

    for (const item of evalSet) {
      const start = performance.now();
      const queryEmbedding = await client.embed(item.query);
      const docEmbeddings = await client.embedBatch(item.relevantDocs);
      const latency = performance.now() - start;

      const similarities = docEmbeddings.map(d => cosineSimilarity(queryEmbedding, d));
      const sortedIndices = similarities
        .map((sim, idx) => ({ sim, idx }))
        .sort((a, b) => b.sim - a.sim)
        .slice(0, k)
        .map(x => x.idx);

      const hits = sortedIndices.filter(idx => item.relevantDocs.includes(item.relevantDocs[idx]));
      totalRecall += hits.length / k;
      
      const firstHitIndex = sortedIndices.findIndex(idx => item.relevantDocs.includes(item.relevantDocs[idx]));
      totalMRR += firstHitIndex !== -1 ? 1 / (firstHitIndex + 1) : 0;
      totalLatency += latency;
    }

    results.push({
      modelId,
      recallAtK: totalRecall / evalSet.length,
      mrr: totalMRR / evalSet.length,
      avgLatencyMs: totalLatency / evalSet.length,
    });
  }

  return results.sort((a, b) => b.recallAtK - a.recallAtK);
}

Step 3: Implement Abstraction Layer

Decouple your application logic from the embedding provider. This enables swapping models based on evaluation results.

// embedding-provider.ts
export interface EmbeddingProvider {
  embed(text: string): Promise<Float32Array>;
  embedBatch(texts: string[]): Promise<Float32Array[]>;
  getDimension(): number;
}

export class OllamaEmbeddingProvider implements EmbeddingProvider {
  private baseUrl: string;
  private model: string;
  private dimension: number;

  constructor(config: { baseUrl: string; model: string; dimension: number }) {
    this.baseUrl = config.baseUrl;
    this.model = config.model;
    this.dimension = config.dimension;
  }

  async embed(text: string): Promise<Float32Array> {
    const response = await fetch(`${this.baseUrl}/api/embed`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ model: this.model, input: text }),
    });
    const data = await response.json();
    return new Float32Array(data.embeddings[0]);
  }

  // ... embedBatch implementation ...
  getDimension(): number { return this.dimension; }
}

Step 4: Architecture Decisions

• Normalization: Ensure your vector database and embedding model agree on normalization. Most cosine similarity searches require L2-normalized vectors. Models like nomic-embed-text output normalized vectors by default; others require explicit normalization.
• Quantization: For local deployments, use 4-bit quantized versions of embedding models (e.g., nomic-embed-text.Q4_K_M) to reduce VRAM usage by 75% with negligible accuracy loss.
• Caching: Implement a semantic cache for embeddings if your knowledge base is static. This eliminates redundant inference costs.

Pitfall Guide

1. Ignoring Normalization Requirements:

   • Mistake: Feeding raw embeddings into a vector store configured for cosine similarity without L2 normalization.
   • Impact: Retrieval quality collapses; distance metrics become meaningless.
   • Fix: Verify model documentation. If the model does not output normalized vectors, apply v = v / ||v|| before storage.

2. Dimensionality Mismatch During Model Swap:

   • Mistake: Switching from a 768-dimension model to a 1024-dimension model without re-indexing the vector store.
   • Impact: Vector store throws schema errors or truncates vectors, causing data corruption.
   • Fix: Always plan for full re-indexing when changing models. Store dimension in your configuration and validate against the vector store schema at startup.

3. Over-Indexing on MTEB Leaderboards:

   • Mistake: Selecting a model because it ranks #1 on MTEB without testing on your data.
   • Impact: Poor retrieval performance due to domain shift.
   • Fix: MTEB is a proxy, not a guarantee. Always run domain-specific recall evaluations.

4. Latency Blindness in Real-Time Applications:

   • Mistake: Using a large embedding model (e.g., 7B parameter encoder) for a low-latency chatbot interface.
   • Impact: User-facing latency increases by hundreds of milliseconds, degrading UX.
   • Fix: Profile p95 latency on your hardware. Use smaller, efficient models (e.g., nomic-embed-text, BGE-small) for latency-sensitive paths.

5. Chunking Strategy Misalignment:

   • Mistake: Using a model optimized for long contexts with very small chunks, or vice versa.
   • Impact: Wasted compute or loss of semantic context.
   • Fix: Align chunk size with model context window and task. For retrieval, chunks of 256-512 tokens often yield better granularity than full-document embeddings.

6. Multi-lingual Assumptions:

   • Mistake: Using an English-only model for a multilingual corpus.
   • Impact: Complete failure to retrieve non-English documents.
   • Fix: Use multilingual models like BGE-M3 or e5-mistral for global datasets.

7. Cost Creep in Commercial APIs:

   • Mistake: Scaling a RAG system with high token volume using commercial embedding APIs without budget caps.
   • Impact: Unexpected infrastructure costs.
   • Fix: For high-volume production, evaluate self-hosted open-source models. The cost of a single GPU often pays for itself within weeks compared to API fees.

Production Bundle

Action Checklist

• Audit Retrieval Failures: Analyze logs to identify queries returning irrelevant results; these define your evaluation set.
• Create Domain Eval Set: Compile 500-1000 query-document pairs representative of production traffic.
• Benchmark Candidates: Run nomic-embed-text, BGE-M3, and text-embedding-3-small against the eval set. Measure Recall@5 and Latency.
• Check Normalization: Verify if selected model requires L2 normalization and update preprocessing pipeline.
• Validate Dimensions: Ensure vector store schema matches model dimensionality; schedule re-index if changing models.
• Profile Latency: Test p95 inference latency on target hardware; apply quantization if VRAM or latency is constrained.
• Implement Abstraction: Wrap embedding calls in a provider interface to allow future model swaps.
• Monitor Drift: Set up alerts for embedding distribution shifts if data sources change frequently.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Enterprise RAG (Sensitive Data)	Self-hosted nomic-embed-text	Zero data leakage; high performance; low latency.	Low (GPU amortization).
Multi-lingual Support	Self-hosted BGE-M3	Handles 100+ languages; long context; strong cross-lingual retrieval.	Medium (Higher VRAM).
Low-Latency Chatbot	Quantized BGE-small	Sub-10ms latency; sufficient accuracy for conversational retrieval.	Low.
Niche Domain (e.g., Legal)	Fine-tuned BGE	Domain adaptation boosts recall by 15%+ over general models.	High (Fine-tuning dev cost).
Prototype / Low Volume	text-embedding-3-small API	Zero infra; fast integration; good general quality.	Variable (Per-token cost).

Configuration Template

Ready-to-use TypeScript configuration for a local embedding service using Ollama.

// config/embedding-config.ts
export interface EmbeddingConfig {
  provider: 'ollama' | 'openai' | 'custom';
  model: string;
  dimension: number;
  normalize: boolean;
  batchSize: number;
  timeoutMs: number;
}

export const LOCAL_EMBEDDING_CONFIG: EmbeddingConfig = {
  provider: 'ollama',
  // nomic-embed-text is optimized for local retrieval
  model: 'nomic-embed-text', 
  dimension: 768,
  // nomic-embed-text outputs normalized vectors; set to false if using raw models
  normalize: false, 
  batchSize: 32,
  timeoutMs: 5000,
};

// docker-compose.yml snippet for Ollama
/*
version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

volumes:
  ollama_data:
*/

Quick Start Guide

Get a local embedding model running and tested in under 5 minutes.

1. Install Ollama:

   curl -fsSL https://ollama.com/install.sh | sh
   

2. Pull Embedding Model:

   ollama pull nomic-embed-text
   

3. Verify Embeddings:

   curl http://localhost:11434/api/embed -d '{
     "model": "nomic-embed-text",
     "input": "Embedding models convert text to vectors."
   }'
   

   Expected output: JSON with embeddings array containing 768 floats.

4. Integrate in TypeScript:

   import { createClient } from 'ollama';
   const ollama = createClient({ host: 'http://localhost:11434' });
   const result = await ollama.embed({ model: 'nomic-embed-text', input: 'Test' });
   console.log(`Dimension: ${result.embeddings[0].length}`);
   

5. Run Benchmark: Use the benchmark-runner.ts provided in the Core Solution to compare nomic-embed-text against your current model on your data.