Candidate Generation at Scale for Large Catalogs

using cosine similarity, apply L2 normalization to vectors before indexing. If using maximum inner product search (MIPS), preserve raw magnitudes but ensure the index supports inner-product distance metrics.

2. Index Topology and Sharding

Avoid monolithic indices. Shard the catalog based on business dimensions that naturally partition the search space, such as region, content type, or popularity strata. Sharding reduces the search space per query, lowers memory pressure, and allows different index configurations per shard.

For example, high-traffic "head" items can use a graph-based index (HNSW) for maximum recall, while long-tail items can use a compressed IVF+PQ index to save memory. The retrieval service routes queries to the appropriate shards based on the query context.

3. The Retrieval Cascade

Implement a cascade of filters that progressively reduce the candidate set while protecting recall. The cascade order is critical for latency:

1. Hard Filters: Apply deterministic business rules (availability, permissions, blacklists) before any vector search.
2. Taxonomy/Bucket Filter: Narrow candidates by category or price range using inverted indices.
3. Coarse ANN Query: Query the ANN index with conservative parameters (e.g., low nprobe or efSearch) to retrieve a superset.
4. Pre-Ranking: Apply a lightweight model (MLP or boosted tree) to score and prune candidates to a manageable size (e.g., 50–200 items).
5. Final Ranking: Pass the pruned set to the heavy ranker for final scoring.

4. Freshness Mechanism: Hot/Cold Split

To balance offline breadth with online freshness, maintain two index layers:

• Cold Index: A compressed, batch-updated index containing the full catalog. Rebuilt periodically (e.g., daily).
• Hot Index: A lightweight, in-memory index for recently added or updated items. Updated in real-time via streaming events.

The retrieval service queries both indices and merges results, ensuring fresh items are surfaced without requiring full index rebuilds.

Implementation Example: Retrieval Orchestrator

The following TypeScript example demonstrates a modular retrieval service that implements sharding, caching, and the cascade logic. This code uses a generic VectorIndex interface to abstract the underlying ANN library (e.g., FAISS, HNSWLib).

import { VectorIndex, SearchResult } from './vector-index.interface';
import { CacheLayer } from './cache-layer';
import { PreRanker } from './pre-ranker';

interface RetrievalConfig {
  shards: ShardConfig[];
  cacheTTLSeconds: number;
  maxCandidates: number;
  hotIndexTTLSeconds: number;
}

interface ShardConfig {
  name: string;
  index: VectorIndex;
  filter: (item: Item) => boolean;
  weight: number; // For merging results
}

export class RetrievalOrchestrator {
  private config: RetrievalConfig;
  private cache: CacheLayer;
  private preRanker: PreRanker;

  constructor(config: RetrievalConfig, cache: CacheLayer, preRanker: PreRanker) {
    this.config = config;
    this.cache = cache;
    this.preRanker = preRanker;
  }

  async getCandidates(
    userId: string,
    queryVector: number[],
    filters: FilterCriteria
  ): Promise<Candidate[]> {
    // 1. Check L2 Cache for precomputed candidates
    const cacheKey = `cand:${userId}:${this.computeFilterHash(filters)}`;
    const cached = await this.cache.get(cacheKey);
    if (cached) {
      return cached.slice(0, this.config.maxCandidates);
    }

    // 2. Execute Cascade: Hard Filters -> ANN -> Pre-Rank
    const candidates: Candidate[] = [];

    // Query each shard concurrently
    const shardPromises = this.config.shards.map(async (shard) => {
      // Apply shard-specific filter
      const filteredQuery = this.applyShardFilter(queryVector, shard.filter);
      if (!filteredQuery) return [];

      // ANN Search with conservative parameters
      const results = await shard.index.search(filteredQuery, 200);
      return results.map(res => ({
        itemId: res.id,
        score: res.score * shard.weight,
        shard: shard.name
      }));
    });

    const shardResults = await Promise.all(shardPromises);
    const rawCandidates = shardResults.flat();

    // 3. Pre-Ranking Stage
    const scoredCandidates = await this.preRanker.score(rawCandidates, filters);

    // 4. Sort and Prune
    const topCandidates = scoredCandidates
      .sort((a, b) => b.score - a.score)
      .slice(0, this.config.maxCandidates);

    // 5. Populate Cache with TTL Jitter to prevent thundering herd
    const jitter = Math.floor(Math.random() * 10);
    await this.cache.set(cacheKey, topCandidates, this.config.cacheTTLSeconds + jitter);

    return topCandidates;
  }

  private computeFilterHash(filters: FilterCriteria): string {
    // Deterministic hash of filter criteria for cache key
    return JSON.stringify(filters).hashCode();
  }

  private applyShardFilter(vector: number[], filter: (item: Item) => boolean): number[] | null {
    // Logic to determine if query matches shard criteria
    // Returns vector if match, null otherwise
    return filter({} as Item) ? vector : null;
  }
}

Architecture Decisions:

• Concurrent Shard Queries: Using Promise.all ensures that sharding does not add serial latency. Each shard is queried in parallel, and results are merged afterward.
• Cache Jitter: Adding random jitter to cache TTLs prevents cache stampedes when popular keys expire simultaneously.
• Weighted Merging: Shards can have different weights to prioritize certain categories or regions during result merging.
• Pre-Ranker Integration: The pre-ranker is applied after ANN retrieval to reduce the candidate set before the heavy ranker, optimizing the latency budget.

Pitfall Guide

Production retrieval systems fail due to operational oversights, not algorithmic flaws. The following pitfalls are common in large-scale deployments.

Pitfall Name	Explanation	Fix
Aggregate Recall Trap	Optimizing for global recall@k masks poor performance on tail items. Heavy quantization (e.g., 4-bit PQ) often degrades tail recall significantly while preserving head recall.	Segment evaluation by popularity buckets (head/mid/tail). Monitor tail recall separately and adjust quantization parameters per shard.
Vector Caching Anti-Pattern	Storing raw embedding vectors in distributed caches (e.g., Redis) increases memory usage and network overhead. Vectors are large and rarely needed outside the ANN node.	Cache only candidate IDs or pre-ranked results. Persist vectors on the ANN serving nodes and use local memory for vector storage.
Index Rebuild Churn	Rebuilding indices from scratch causes downtime and high CPU spikes. Incremental updates to graph-based indices (HNSW) can be expensive and lead to fragmentation.	Implement shadow indices: build the new index in the background and atomically swap it with the live index. Use periodic compaction for graph indices.
Dimensionality Bloat	Using high-dimensional embeddings (e.g., 1024+) without justification increases index size and degrades quantization performance.	Calibrate dimensionality against recall loss. Use 64–256 dimensions for most workloads. Apply dimensionality reduction (PCA) if necessary before indexing.
Thundering Herd on Miss	Cache expiration for popular queries causes a spike in ANN queries, overwhelming the index and increasing latency.	Implement request coalescing to batch simultaneous queries for the same key. Use cache jitter and background refresh patterns.
Ignoring Update Cadence	Item embeddings may become stale due to price changes, availability, or content updates. A daily rebuild may be too slow for dynamic catalogs.	Maintain a hot/cold index split. Update the hot index in real-time via streaming events. Set update cadence based on business dynamics (e.g., hourly for dynamic catalogs).
Pruning Cascade Misordering	Applying expensive filters or ANN queries before hard filters wastes latency budget.	Order the cascade by cost: Hard filters → Taxonomy filters → Coarse ANN → Pre-ranker → Heavy ranker. Ensure each stage reduces the candidate set significantly.

Production Bundle

Action Checklist

• Segment recall evaluation by item popularity (head/mid/tail) and user activity level.
• Implement a hot/cold index split to support real-time freshness without full rebuilds.
• Configure L1/L2 caching with TTL jitter and request coalescing to prevent cache stampedes.
• Design the retrieval cascade to apply hard filters before ANN queries.
• Calibrate embedding dimensionality and quantization parameters per shard based on tail recall metrics.
• Implement shadow index swaps for zero-downtime index updates.
• Monitor p99 latency and tail recall continuously; set alerts for degradation.
• Use α-NDCG or Intra-List Similarity (ILS) to measure diversity and avoid redundancy in candidate sets.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time Personalization	Hybrid Cascade + Hot/Cold Split	Balances low latency with fresh, diverse recommendations.	Medium (Requires hot index infrastructure).
Static Catalog (Low Churn)	Monolithic HNSW or Sharded IVF+PQ	Simpler architecture; updates are infrequent.	Low (Reduced operational complexity).
Memory-Constrained Environment	Sharded IVF+PQ with Aggressive Quantization	Compresses vectors to fit within memory limits.	Low (Savings on RAM), but monitor tail recall.
GPU-Available Cluster	FAISS with GPU Acceleration	Accelerates batched retrievals and large index builds.	High (GPU costs), but improves throughput.
High Tail-Recall Requirement	Sharded HNSW with Per-Shard Tuning	Preserves recall for long-tail items by avoiding heavy quantization.	Medium (Higher memory per shard).

Configuration Template

The following YAML template defines a retrieval service configuration with sharding, caching, and cascade parameters. This can be used to deploy the orchestrator in a production environment.

retrieval:
  max_candidates: 100
  cache_ttl_seconds: 60
  hot_index_ttl_seconds: 300

  shards:
    - name: "electronics-head"
      index_type: "HNSW"
      index_params:
        m: 16
        ef_construction: 200
        ef_search: 50
      filter: "category == 'electronics' AND popularity > 0.8"
      weight: 1.2

    - name: "electronics-tail"
      index_type: "IVF_PQ"
      index_params:
        nlist: 1024
        m: 8
        nbits: 8
        nprobe: 32
      filter: "category == 'electronics' AND popularity <= 0.8"
      weight: 1.0

    - name: "hot-items"
      index_type: "FLAT"
      index_params: {}
      filter: "updated_at > now() - 1h"
      weight: 1.5

  cascade:
    hard_filters:
      - "availability == true"
      - "region == user_region"
    pre_ranker:
      model: "mlp-v2"
      max_candidates: 200

Quick Start Guide

1. Define Embedding Schema: Train a two-tower model and export item embeddings with consistent normalization. Store embeddings in a versioned object store.
2. Deploy Index Service: Spin up the ANN serving nodes with the configuration template. Initialize shards and load embeddings from the object store.
3. Configure Hot/Cold Split: Set up a streaming pipeline to ingest item updates and populate the hot index. Configure the cold index to rebuild periodically.
4. Integrate Retrieval Client: Implement the retrieval client in your application service. Use the orchestrator to query candidates, apply filters, and cache results.
5. Monitor and Tune: Deploy monitoring for p99 latency, tail recall, and cache hit rates. Adjust nprobe, ef_search, and shard weights based on observed performance. Iterate on quantization parameters to balance memory and recall.