Incremental Vector Indexing: Decoupling RAG Updates from Corpus Size

Current Situation Analysis

The modern RAG ecosystem is built around a fundamental illusion: that knowledge bases are static archives. Frameworks, tutorials, and quickstart guides consistently demonstrate the same linear workflow: ingest documents, chunk text, generate embeddings, load vectors, and query. This "happy path" works flawlessly for proof-of-concepts and demo environments. It completely collapses when applied to production systems where documentation, policies, or domain knowledge evolve daily.

The industry overlooks this gap because vector database abstractions hide state management behind simple add() and query() methods. Developers assume that updating a document triggers an automatic delta sync. In reality, most open-source and lightweight vector stores treat the index as an append-only log. When source files change, the naive implementation is to wipe the index and re-embed the entire corpus. On cloud infrastructure with dedicated GPUs, this is an inconvenience. On constrained edge hardware—CPU-only machines with under 4 GB of RAM—it is a critical failure point.

Embedding models like nomic-embed-text are computationally expensive. Running them sequentially across dozens or hundreds of documents on a Raspberry Pi-class device consumes minutes of CPU time and saturates memory bandwidth. If a single paragraph is corrected in a 50-document survival guide, a full re-index forces the system to reprocess 49 unchanged files. This creates a feedback loop that kills development velocity: engineers stop updating knowledge bases because the indexing latency breaks their workflow. The problem isn't the embedding model; it's the lack of a deterministic, low-overhead change detection layer between the filesystem and the vector store.

WOW Moment: Key Findings

The breakthrough comes from treating document updates as a state synchronization problem rather than a data processing problem. By introducing a content-fingerprinting layer, indexing time shifts from being proportional to total corpus size to being proportional to the change delta.

Approach	Processing Duration	CPU Cycles Wasted	Memory Pressure	Update Latency
Full Corpus Re-Embedding	~6 minutes (47 docs)	100% of corpus	High (reloads all chunks)	Linear with corpus size
SHA-256 Delta Indexing	~40 seconds (3 changed docs)	~6% of corpus	Low (streams only deltas)	Constant relative to change rate

This finding matters because it transforms RAG from a batch archive into a responsive, living system. Developers can iterate on knowledge bases in real-time without waiting for background jobs to complete. More importantly, it enables offline-first architectures where compute resources are strictly bounded. The indexing pipeline no longer competes with inference workloads for CPU cycles, allowing qwen2.5:3b via Ollama to maintain low query latency even during active knowledge base updates.

Core Solution

The architecture relies on three decoupled components: a streaming content hasher, a state manifest, and a vector lifecycle manager. Each component handles a specific responsibility, ensuring that the system remains predictable under resource constraints.

Step 1: Streaming Content Fingerprinting

File metadata like modification timestamps (mtime) or file sizes are unreliable for change detection. Deployment scripts, version control operations, and backup tools frequently update timestamps without altering content. A single character change in a 10 KB file leaves the size unchanged. Cryptographic hashing solves this by generating a deterministic fingerprint based purely on byte content.

SHA-256 is preferred over MD5 not because collision resistance is critical here, but because it provides a standardized, widely audited algorithm with negligible performance overhead at this scale. The implementation must stream the file in fixed-size chunks to prevent memory exhaustion when processing large documents.

import { createHash } from 'node:crypto';
import { createReadStream } from 'node:fs';
import { promisify } from 'node:util';

const CHUNK_SIZE = 8192;

export async function computeContentHash(filePath: string): Promise<string> {
  const hash = createHash('sha256');
  const stream = createReadStream(filePath, { highWaterMark: CHUNK_SIZE });
  
  return new Promise((resolve, reject) => {
    stream.on('data', (chunk: Buffer) => hash.update(chunk));
    stream.on('end', () => resolve(hash.digest('hex')));
    stream.on('error', reject);
  });
}

Streaming in 8 KB chunks ensures the memory footprint remains flat regardless of document size. This is critical for sub-4 GB RAM environments where loading multi-megabyte markdown files into heap memory would trigger garbage collection spikes or OOM kills.

Step 2: State Manifest Architecture

The vector store needs to know which files were previously indexed, their cryptographic fingerprints, and the exact vector IDs assigned to their chunks. A lightweight JSON manifest serves as the source of truth. It maps file paths to their hash, assigned vector identifiers, and last sync timestamp.

export interface DocumentState {
  hash: string;
  vectorIds: number[];
  lastIndexed: string;
}

export interface IndexManifest {
  version: number;
  documents: Record<string, DocumentState>;
}

Tracking vectorIds per document is non-negotiable. Without it, the system cannot cleanly remove stale vectors when a file is updated or deleted. The manifest acts as a bridge between the filesystem and the vector index, enabling precise lifecycle operations.

Step 3: Delta Resolution Logic

The indexer compares the current filesystem state against the manifest. The resolution logic follows a strict decision tree:

1. Hash matches stored hash → Document unchanged. Skip embedding.
2. Hash differs → Document modified. Invalidate old vectors, re-embed, update manifest.
3. Path exists in manifest but not filesystem → Document deleted. Remove vectors, purge manifest entry.
4. Path exists in filesystem but not manifest → New document. Embed fresh, record in manifest.

This logic ensures that only the delta crosses the embedding pipeline. The computational cost becomes a function of change frequency, not total knowledge base size.

Step 4: Vector Lifecycle Management

FAISS does not support native vector deletion. Attempting to remove individual vectors requires rebuilding the index from the remaining vectors. For small to medium corpora (<10,000 vectors), this rebuild is fast enough to remain transparent. For larger deployments, the architecture must migrate to a vector database with native delete semantics (e.g., Qdrant, Weaviate, or Milvus).

The lifecycle manager handles three operations:

• Invalidate: Collect vector IDs from the manifest for modified/deleted files.
• Rebuild: Filter out invalidated IDs, reconstruct the FAISS index from surviving vectors.
• Insert: Append new embeddings, update manifest with new vector IDs.

export class VectorLifecycleManager {
  constructor(private index: faiss.IndexFlatIP, private manifest: IndexManifest) {}

  async applyDelta(
    newEmbeddings: number[][],
    newIds: number[],
    obsoleteIds: number[]
  ): Promise<void> {
    if (obsoleteIds.length > 0) {
      await this.rebuildIndex(obsoleteIds);
    }
    if (newEmbeddings.length > 0) {
      this.index.add(newEmbeddings);
      this.manifest.lastVectorId = Math.max(...newIds);
    }
  }

  private async rebuildIndex(obsoleteIds: number[]): Promise<void> {
    const survivingVectors = this.extractSurvivingVectors(obsoleteIds);
    this.index.reset();
    this.index.add(survivingVectors);
  }
}

Architecture Rationale:

• SHA-256 provides deterministic matching with microsecond overhead.
• JSON manifest prioritizes portability and simplicity over ACID guarantees. Suitable for single-process indexing.
• FAISS rebuild strategy accepts a known limitation in exchange for zero external dependencies. Matches the offline-first, CPU-only constraint.
• Decoupled components allow swapping the vector store or embedding model without rewriting the delta detection logic.

Pitfall Guide

1. Relying on Filesystem Metadata for Change Detection

Explanation: Using mtime, ctime, or file size as a change trigger causes false positives. Backup tools, IDEs, and deployment scripts routinely update timestamps. A file copied via rsync or extracted from a ZIP archive will show a new timestamp despite identical content. Fix: Always hash the raw byte stream. Metadata is useful for logging, never for state comparison.

2. Hashing Chunks Instead of Source Files

Explanation: Generating hashes per text chunk creates a mismatch when chunking strategies change. If you adjust chunk size or overlap, the same document produces different chunk boundaries, triggering unnecessary re-embeddings. Fix: Hash the source file before chunking. Store the file-level hash in the manifest. Chunk boundaries are a transformation layer, not a state layer.

3. Ignoring FAISS Deletion Semantics

Explanation: FAISS indexes are append-only by design. Calling remove() on a non-existent method or attempting to patch the index file corrupts the binary structure. Developers often assume vector databases behave like relational tables. Fix: Implement a rebuild strategy for small indices. For indices >10k vectors, migrate to a managed vector store with native deletion. Document the threshold clearly in architecture reviews.

4. Manifest Concurrency Collisions

Explanation: A single JSON file has no locking mechanism. If two indexing processes run simultaneously (e.g., a cron job and a manual trigger), they will read stale state, overwrite each other's updates, and corrupt the vector ID mapping. Fix: Use file-level locking (flock on Linux) or migrate to SQLite for manifest storage. In single-process deployments, enforce a mutex or PID lock file before indexing begins.

5. Over-Indexing Cosmetic Changes

Explanation: SHA-256 detects byte-level differences. Renaming a section header, fixing a typo, or reformatting whitespace changes the hash and triggers a full re-embed. This is technically correct but can waste compute on semantically identical content. Fix: Accept byte-level hashing as the baseline. If semantic drift detection is required, implement a secondary pass using a lightweight similarity model or LLM-based diff evaluator before committing to re-embedding.

6. Vector ID Space Exhaustion

Explanation: FAISS assigns sequential integer IDs. If you rebuild the index repeatedly without resetting the ID counter, you may eventually hit integer limits or create sparse ID gaps that complicate metadata mapping. Fix: Reset the ID counter during full rebuilds. Use a monotonic counter in the manifest and explicitly map new embeddings to contiguous ID ranges.

7. Skipping Manifest Backups

Explanation: The manifest is the only link between source files and vector IDs. If it corrupts or is deleted, the vector index becomes an opaque blob with no way to trace which vectors belong to which documents. Fix: Implement atomic writes (write to .tmp, rename to .json). Maintain a rolling backup of the last 3 manifest versions. Validate manifest integrity before every indexing run.

Production Bundle

Action Checklist

• Implement streaming SHA-256 hashing with 8 KB chunks to prevent memory spikes
• Design a manifest schema that tracks file path, content hash, vector IDs, and sync timestamp
• Build a delta resolver that compares current filesystem state against the manifest
• Implement FAISS rebuild logic for vector invalidation, with clear scaling thresholds
• Add file-level locking or PID guards to prevent concurrent indexing collisions
• Validate manifest integrity before each run and maintain atomic write backups
• Document the migration path to native-delete vector stores when corpus exceeds 10k vectors

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
<500 documents, CPU-only, offline	SHA-256 delta + FAISS rebuild	Zero external dependencies, predictable memory, fast enough for delta sizes	Near-zero infrastructure cost
500–10,000 documents, single node	SHA-256 delta + SQLite manifest + FAISS	SQLite adds concurrency safety and queryability without heavy overhead	Low (SQLite is embedded)
>10,000 documents, multi-node	Managed vector DB (Qdrant/Weaviate) + content hashing	Native deletion, distributed sync, horizontal scaling	Moderate to high (managed service or cluster ops)
Real-time collaborative editing	Semantic diff layer + chunk-level hashing	Byte-level hashing triggers too often on live docs; semantic filtering reduces false positives	Higher compute cost, lower indexing waste

Configuration Template

indexer:
  source_dir: "./knowledge-base"
  manifest_path: "./data/index_manifest.json"
  vector_store:
    type: "faiss"
    metric: "inner_product"
    rebuild_threshold: 10000
  hashing:
    algorithm: "sha256"
    chunk_size_bytes: 8192
  concurrency:
    max_processes: 1
    lock_file: "./data/indexer.lock"
  embedding:
    model: "nomic-embed-text"
    provider: "ollama"
    batch_size: 32
  lifecycle:
    backup_count: 3
    atomic_write: true
    validate_before_run: true

Quick Start Guide

1. Initialize the manifest: Run the indexer against an empty knowledge base. It will scan all files, compute SHA-256 hashes, generate embeddings, and write the initial index_manifest.json.
2. Verify delta detection: Modify a single markdown file. Run the indexer again. Observe that only the changed file crosses the embedding pipeline. Check logs for hash_match: false and vectors_updated: 1.
3. Test deletion handling: Remove a file from the source directory. Run the indexer. Confirm that the manifest entry is purged and the corresponding vectors are removed via FAISS rebuild.
4. Scale validation: Add 50 new files. Run the indexer. Verify that processing time remains proportional to the 50 new files, not the total corpus size. Monitor CPU and memory to ensure streaming hashing prevents spikes.
5. Deploy to constrained hardware: Transfer the compiled indexer and manifest to your target device. Ensure Ollama and nomic-embed-text are available. Run a full sync, then validate query latency against qwen2.5:3b during active indexing.