How We Built a 3-Tier Node.js Cache That Hits 2.8M ops/sec (And Never Drops a Key on Eviction)

via Count-Min Sketch

Standard LRU (Least Recently Used) caching fails catastrophically during sudden scan floods or data bursts, where long-resident, high-value keys get wiped out by single-access keys.

tricache implements a hybrid eviction algorithm that scores keys based on a combination of Priority Score + Access Frequency + Remaining TTL. To track access frequency without introducing a massive memory footprint, we integrated a Count-Min Sketch using a tiny 4 KB Uint16Array (4 rows × 512 columns).

Every time a key is queried or set, it passes through the sketch. When L1 needs to evict data, it uses reservoir sampling to pick an eviction candidate in a single $O(1)$ pass, ensuring a 78% survival rate for high-frequency keys during benchmark flood tests.

2. Inlined WASM Bloom Filter

Before hitting a synchronous Map lookup or checking the filesystem/Redis on a cache miss, queries pass through an ultra-lightweight WASM Bloom filter.

The filter is optimized with $k=7$ hash probes. At a rated capacity of roughly 18,000 entries, it maintains a false-positive rate of just 1%. If the filter says a key does not exist, it is a guaranteed miss, allowing tricache to skip heavy lookups entirely. The 562-byte WASM binary is fully inlined as a Base64 string—meaning zero filesystem access overhead at boot time.

3. Native Thundering-Herd Prevention

When a highly contested key expires under heavy traffic, hundreds of concurrent requests usually hit the underlying database at the exact same moment.

tricache avoids this with a native Inflight Promise Registry. No matter how many concurrent callers request an expired or missing key, the user-supplied fetchFn fires exactly once. All other concurrent requests hook into that single pending promise, protecting your database or upstream APIs from collapsing under sudden load.

4. Enterprise-Grade Resiliency Out of the Box

• Stale-While-Revalidate (SWR) & Stale-If-Error: Instantly serves stale data to the client while revalidating the data asynchronously in the background. If the upstream database fails during revalidation, staleIfError kicks in to extend the stale entry's life, keeping your app fully operational during upstream outages.
• OOM Guard: A background timer polls heapUsed and heapTotal. If memory utilization breaches a set threshold (e.g., 85%), an emergency eviction routine triggers immediately, clearing out the coldest 20% of L1 memory before the Node.js process crashes.
• At-Rest Encryption: Supports automated AES-256-GCM authenticated encryption for L2 values, disk spill files, and cold-start snapshots, complete with zero-downtime key rotation support.

---

Performance Benchmarks

All metrics were captured executing on a single Node.js thread (with synchronous paths completely un-awaited):

L1 Memory Cache Throughput

Operation

Throughput

Latency

Mechanics

get (Hot Hit)

2.81 M/s

~356 ns

Bloom filter → Map lookup → Return

get (Cold Miss)

7.14 M/s

~140 ns

Bloom filter gates early return

delete (Exact Key)

5.36 M/s

~186 ns

Synchronous Map deletion

End-to-End Service Performance

Operation

Throughput

Latency

get (L1 Warm Hit)

2.03 M/s

~491 ns

get (SWR Stale Serve)

1.78 M/s

~562 ns

get (Miss + fetchFn)

13.7 K/s

~73 µs

---

Quick Start: Getting Started in 30 Seconds

Getting started requires zero initial configuration. Sensible production defaults apply automatically.

npm install tricache
# or pnpm add tricache

Enter fullscreen mode Exit fullscreen mode

import { CacheService } from 'tricache';

// Initialize the process-level singleton
const cache = CacheService.create({
  namespace: 'my-app',
  redisHost: process.env.REDIS_HOST, // Omit in dev to automatically fallback to L1/Disk only
});

// Fetch-or-Get with an automatic database fallback and a 5-minute TTL
const user = await cache.get(
  `user:${userId}`,
  () => db.users.findById(userId),
  300, // TTL in seconds
  { swr: 30 } // Seamless Stale-While-Revalidate for 30 seconds
);

Enter fullscreen mode Exit fullscreen mode

Advanced Group Invalidation with Tags

You can tag items on creation and invalidate entire collections across L1, local disk, and distributed Redis nodes simultaneously:

// Add items to the cache with descriptive tags
await cache.set(`product:1`, productData, 300, undefined, { tags: ['catalog'] });
await cache.set(`product:2`, alternativeData, 300, undefined, { tags: ['catalog'] });

// Atomically evict all catalog entries everywhere
await cache.invalidateTag('catalog');

Enter fullscreen mode Exit fullscreen mode

---

Open Source & Contributing

tricache is fully open-source, licensed under the MIT license, and built entirely using TypeScript. If you are building high-performance Node.js services or want to dive into the codebase (especially src/smart-memory-cache.ts), check out the repository!

👉 Check out the repo on GitHub: github.com/Kareem411/TriCache

Feedback, bug reports, and performance optimizations via Pull Requests are always welcome! Let me know what caching bottlenecks you're currently facing in your architecture below.