docker-compose.yml

Capture (CDC) to stream database changes to a message queue, ensuring index freshness. 2. Separation of Concerns: The search index should be a write-through cache of the source of truth, never the source of truth itself. 3. Reranking Layer: For high-relevance requirements, implement a lightweight reranker service. This decouples the heavy inference logic from the low-latency retrieval path.

Step-by-Step Implementation

1. Schema Definition and Indexing

Define a schema that supports faceting, filtering, and sortable fields. Avoid indexing fields that are only used for display.

// search-schema.ts
import { TypesenseClient } from 'typesense';

const client = new TypesenseClient({
  nodes: [{ host: 'search-api', port: 8108, protocol: 'http' }],
  apiKey: process.env.TYPESENSE_API_KEY,
});

export const productsSchema = {
  name: 'products',
  fields: [
    { name: 'title', type: 'string', facet: false },
    { name: 'description', type: 'string', facet: false, stem: true },
    { name: 'category', type: 'string', facet: true },
    { name: 'price', type: 'float', facet: true, optional: true },
    { name: 'rating', type: 'float', facet: false, sort: true },
    { name: 'embedding', type: 'float[]', embed: { from: ['title', 'description'], model_config: { model_name: 'ts/all-MiniLM-L6-v2' } } }
  ],
  default_sorting_field: 'rating'
};

2. Ingestion Pipeline

Implement a resilient ingestion worker that handles upserts and deletes. Use bulk operations to maximize throughput.

// ingestion-worker.ts
import { BulkOperation } from 'typesense';

export async function indexProducts(products: Product[]) {
  const operations: BulkOperation[] = products.map(p => ({
    upsert: {
      id: p.id.toString(),
      title: p.title,
      description: p.description,
      category: p.category,
      price: p.price,
      rating: p.rating,
      // Embeddings can be generated client-side or handled by the search engine
      embedding: await generateEmbedding(`${p.title} ${p.description}`)
    }
  }));

  try {
    await client.collections('products').documents().upsert(operations);
  } catch (error) {
    // Implement retry logic with exponential backoff
    console.error('Indexing failed:', error);
    throw error;
  }
}

3. Query Processing and Hybrid Search

Construct queries that combine keyword matching with semantic filtering. Use the filter_by parameter for exact matches to reduce the search space before ranking.

// search-service.ts
export interface SearchParams {
  query: string;
  filters?: Record<string, string>;
  limit?: number;
  offset?: number;
}

export async function performSearch(params: SearchParams) {
  const { query, filters = {}, limit = 20, offset = 0 } = params;

  // Build filter string dynamically
  const filterString = Object.entries(filters)
    .map(([key, value]) => `${key}:=${value}`)
    .join(' && ');

  const searchParameters = {
    q: query,
    query_by: 'title,description',
    filter_by: filterString || 'true',
    sort_by: 'rating:desc',
    per_page: limit,
    page: (offset / limit) + 1,
    highlight_full_fields: ['description'],
    // Enable vector search if embedding is configured
    vector_query: `embedding:([${await generateEmbedding(query)}])`,
  };

  return client.collections('products').documents().search(searchParameters);
}

4. Reranking Strategy

For production-grade relevance, retrieve a larger candidate set (limit: 50) and apply a reranker to the top results. This corrects BM25's lack of semantic context.

// reranker.ts
export async function rerankResults(query: string, results: any[]) {
  // Call external reranker API or local model
  const reranked = await rerankerService.rank(query, results.map(r => r.title));
  
  // Reorder original results based on reranker scores
  return results.sort((a, b) => {
    const scoreA = reranked.findIndex(r => r.id === a.id);
    const scoreB = reranked.findIndex(r => r.id === b.id);
    return scoreA - scoreB;
  });
}

Pitfall Guide

1. Ignoring Tokenization and Language Nuances

Mistake: Using default tokenization for multi-language or domain-specific data. Impact: Queries like "C++" or "Node.js" fail or return irrelevant results due to special character stripping. Best Practice: Configure token separators and delimiters explicitly. Implement language-specific analyzers and stemmers. Create a custom tokenizer for technical terms.

2. Index Bloat and Unnecessary Fields

Mistake: Indexing every column from the database. Impact: Increased memory usage, slower indexing, and degraded query performance. Best Practice: Only index fields used in query_by, filter_by, or sort_by. Store display-only fields in the source database and fetch via join or API call post-search.

3. Latency Spikes on High-Cardinality Filters

Mistake: Applying filters on high-cardinality fields without proper indexing. Impact: Query performance degrades linearly with dataset size. Best Practice: Ensure filtered fields are marked as facet: true. Use numeric types for ranges. Pre-compute derived fields if complex logic is required in filters.

4. Stale Index Synchronization

Mistake: Relying on periodic cron jobs for index updates. Impact: Users see deleted items or outdated prices, causing trust issues. Best Practice: Implement real-time CDC. Use a message queue (Kafka/RabbitMQ) to decouple the database write from the index update. Monitor lag metrics between DB commit and index availability.

5. The "Zero Results" Dead End

Mistake: Returning empty pages without guidance. Impact: User frustration and session termination. Best Practice: Implement query relaxation strategies. If zero results, retry with relaxed constraints (e.g., remove filters, expand fuzzy matching, or fall back to semantic search). Always provide "Did you mean?" suggestions.

6. Security Leaks in Multi-Tenancy

Mistake: Exposing all data to all users via search API. Impact: Data breaches and compliance violations. Best Practice: Use scoped API keys or dynamic filters based on user context. Never trust client-side filtering. Enforce tenant isolation at the query layer using filter_by: tenant_id:=<user_tenant>.

7. Neglecting Query Caching

Mistake: Executing expensive queries repeatedly for popular terms. Impact: Unnecessary compute load and latency. Best Practice: Implement a caching layer for frequent queries. Use time-based invalidation. Cache results based on the normalized query string and filter combination.

Production Bundle

Action Checklist

• Define Schema Strategy: Audit database fields; map only searchable, filterable, and sortable fields to the search schema.
• Implement CDC Pipeline: Set up Debezium or logical replication to stream changes to a message queue; build idempotent upsert workers.
• Configure Synonyms: Create a synonym map for domain terminology and common misspellings; update dynamically via API.
• Set Up Monitoring: Instrument metrics for P95 latency, zero-result rate, and index lag; alert on degradation.
• Implement Query Relaxation: Add logic to fallback to broader searches or semantic queries when exact matches fail.
• Secure Access: Apply scoped API keys; enforce tenant filtering in all search requests; audit access logs.
• Load Test: Simulate traffic spikes; verify performance with large result sets and complex filter combinations.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
MVP / Low Traffic	SQLite + FTS or Meilisearch	Minimal ops overhead; sufficient for <100k docs.	Low
E-commerce / SaaS	Typesense / Meilisearch	Fast setup; excellent relevance tuning; low latency.	Medium
Semantic / RAG App	Hybrid (Vector + BM25)	Required for intent-based search and LLM integration.	High
Regulated Data	Elasticsearch / OpenSearch	Mature security features; on-prem deployment options.	High

Configuration Template

Copy this Docker Compose configuration to bootstrap a Typesense search cluster with monitoring.

# docker-compose.yml
version: '3.8'
services:
  typesense:
    image: typesense/typesense:0.25.1
    restart: always
    ports:
      - "8108:8108"
    volumes:
      - ./data:/data
    command: '--data-dir /data --api-key=CHANGE_ME --enable-cors'
    environment:
      - TYPESENSE_API_KEY=CHANGE_ME

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

Quick Start Guide

1. Initialize Cluster: Run docker-compose up -d to start the search node and monitoring stack.
2. Create Collection: Execute the schema creation script using the TypeScript client provided in the Core Solution.
3. Index Sample Data: Run the ingestion worker against a subset of production data to validate mapping and performance.
4. Test Queries: Use the search service endpoint to run queries; verify relevance and latency using Grafana dashboards.
5. Deploy Workers: Scale the ingestion workers horizontally; configure auto-scaling based on queue depth.