Preventing Cache Stampedes with Singleflight & Multi-Tier Redis: Slashing P99 Latency to 22ms
Eliminating the thundering herd problem on SubsDrop's product catalog endpoints when thousands of concurrent users refresh popular subscription items.
When digital subscription batches like Canva Pro or ChatGPT Plus drop on SubsDrop, thousands of users hit catalog endpoints simultaneously. If a cache key expires, naive systems suffer catastrophic cache stampedes. Here is how we engineered singleflight request deduplication with tiered caching.
Table of Contents (4 sections)▼
1. Anatomy of a Cache Stampede
A cache stampede (or thundering herd problem) occurs when high-concurrency requests hit an expired cache key simultaneously, triggering hundreds of identical database queries at once. Singleflight request coalescing eliminates this by intercepting concurrent in-flight reads, executing exactly one query, and broadcasting the result across all callers.
Imagine a catalog item like 'ChatGPT Plus Subscription' cached in Redis with a 10-minute TTL. While valid, responses return in ~2ms.
At minute 10:01, the key expires. In that exact millisecond, 400 concurrent incoming HTTP requests check Redis, find a cache miss, and all 400 requests trigger identical, heavy MongoDB aggregation queries simultaneously.
This classic 'cache stampede' (or thundering herd) saturates the database connection pool, pegs database CPU to 95%, and causes response latencies to spike from 2ms to 240ms+.
2. The L1 Memory + L2 Redis Architecture
To permanently solve this, we architected a two-tier caching fabric:
• L1 In-Memory LRU Cache: Lives inside each Node.js process with an ultra-short 15-second TTL. This resolves in <0.1ms without any network TCP round-trip to Redis.
• L2 Distributed Redis Cache: Shared across our cluster processes with a 10-minute TTL.
• In-Flight Promise Registry: A thread-safe Promise latch that ensures only one single query executes against the database on a cache miss.
3. Implementing Singleflight in TypeScript
Below is our production-tested singleflight pattern implemented in TypeScript:
import Redis from "ioredis";
import { LRUCache } from "lru-cache";
const redis = new Redis(process.env.REDIS_URL!);
const localLru = new LRUCache<string, any>({
max: 1000,
ttl: 1000 * 15, // 15-second L1 memory cache
});
const inFlightRegistry = new Map<string, Promise<any>>();
export async function fetchTieredWithSingleFlight<T>(
cacheKey: string,
dbFallback: () => Promise<T>,
redisTtlSec = 600
): Promise<T> {
// 1. Check L1 Memory Cache (<0.1ms)
const l1Hit = localLru.get(cacheKey);
if (l1Hit) return l1Hit as T;
// 2. Check L2 Redis Cache (~2ms)
const l2Hit = await redis.get(cacheKey);
if (l2Hit) {
const parsed = JSON.parse(l2Hit);
localLru.set(cacheKey, parsed);
return parsed as T;
}
// 3. Singleflight Latch: If query already in flight, await existing Promise
if (inFlightRegistry.has(cacheKey)) {
return inFlightRegistry.get(cacheKey) as Promise<T>;
}
// 4. Execute single query and share with all concurrent callers
const queryPromise = (async () => {
try {
const freshData = await dbFallback();
await redis.setex(cacheKey, redisTtlSec, JSON.stringify(freshData));
localLru.set(cacheKey, freshData);
return freshData;
} finally {
// Clean up latch once resolved or rejected
inFlightRegistry.delete(cacheKey);
}
})();
inFlightRegistry.set(cacheKey, queryPromise);
return queryPromise;
}4. Benchmarking 2,500 Concurrent Hits
Using autocannon to simulate 2,500 concurrent connections hitting an expired catalog key:
• Without Singleflight: 2,500 DB queries triggered. DB CPU hit 92%. P99 latency: 284ms. 14 timed-out connections.
• With Singleflight: Exactly 1 DB query executed. DB CPU remained under 12%. P99 latency: 22ms. 0 dropped connections.
Review the verified telemetry in the Redis Caching & Singleflight Case Study.
Frequently Asked Questions
- What is a cache stampede in Node.js?
- A cache stampede (thundering herd) occurs when a cached key expires and hundreds of concurrent requests simultaneously trigger identical database queries. In Node.js, this overwhelms the event loop and database connection pool, causing latency spikes from 2ms to 240ms+ and potential service degradation.
- How does singleflight prevent cache stampedes?
- Singleflight (request coalescing) uses an in-flight Promise registry. When a cache miss occurs, only the first request executes the database query. All subsequent concurrent requests for the same key await the same Promise, ensuring exactly one database query per cache miss regardless of concurrency level.
- What is the difference between L1 and L2 cache in Node.js?
- L1 cache is an in-process LRU (Least Recently Used) memory cache inside each Node.js process with ultra-short TTL (~15 seconds), resolving in <0.1ms without network I/O. L2 cache is a distributed Redis cache shared across all processes with longer TTL (~10 minutes), resolving in ~2ms via TCP. Together they eliminate most database queries.
Related Field Notes
Bypassing Safari ITP, iOS 14.5 ATT restrictions, and ad-blockers on MoneTrix by routing e-commerce transactions through server containers with deterministic deduplication.
Out-of-band VNC rescue, terminating masked cryptominers, scrubbing persistent cronhooks, and containerizing production services with zero database loss.