A cache hit is cheap until the key expires. Imagine 800 requests arriving at the same moment for the same popular dashboard. If every miss triggers the same PostgreSQL query, Redis has not protected the database; it has synchronized the load spike. This is a cache stampede, and it often appears during a traffic surge, deployment, or Redis restart.
1. Understand the failure before adding a lock
Suppose a popular result takes 120 ms to compute and normally receives 400 requests per second. During a cold interval, dozens of requests can enter the database path before the first fill completes. If the query uses a scarce connection pool, a single key can starve unrelated endpoints. Cache hit rate alone hides this: graph miss rate, origin concurrency, pool wait time, p95/p99 latency, and the number of concurrent rebuilds per key.
First establish a correctness contract. Is a result allowed to be stale for 30 seconds? Must permissions or inventory be current? A cache policy without a freshness budget is a production incident waiting for a marketing campaign.
2. Spread expirations with TTL jitter
When many keys are populated together, a fixed TTL can cause synchronized expiration. Randomize expiration around a base TTL so their refreshes spread over time.
const baseSeconds = 300;
const jitterSeconds = Math.floor(Math.random() * 61);
await redis.set(key, JSON.stringify(value), {
EX: baseSeconds + jitterSeconds
});
Jitter reduces synchronized expiration across different keys. It does not stop 100 callers from rebuilding the same expired key. For that, coordinate work.
3. Coalesce concurrent requests inside one process
For a single Node.js instance, use a promise map. Every caller of the same key awaits one in-flight fetch. Remove the entry on success and failure, or a rejected promise can poison the key indefinitely.
const inflight = new Map();
async function singleFlight(key, loader) {
if (inflight.has(key)) return inflight.get(key);
const task = Promise.resolve().then(loader);
inflight.set(key, task);
try { return await task; }
finally { if (inflight.get(key) === task) inflight.delete(key); }
}
Use this around the miss path after checking Redis. It reduces duplicate work per process; six Kubernetes replicas can still launch six rebuilds. Bound the number of distinct in-flight keys to prevent an attacker or an accidental high-cardinality query from growing memory without limit.
4. Choose stale-while-revalidate when the product permits it
Store a payload with a logical freshness deadline and a longer physical Redis TTL. During the stale window, serve the previous value immediately while one worker refreshes it. This keeps latency stable when the origin is slow. For personalized or security-sensitive data, scope keys correctly and never serve stale authorization decisions.
Keep the stale window finite. If refresh fails repeatedly, decide explicitly whether to serve a bounded stale response, degrade functionality, or return an error. Do not quietly turn a five-minute cache into an indefinite source of truth.
5. Coordinate across replicas carefully
A short Redis lease can reduce duplicate refreshes across pods:
const token = crypto.randomUUID();
const acquired = await redis.set(
`refresh:${key}`, token, { NX: true, PX: 10000 }
);
if (acquired === 'OK') {
try {
const fresh = await loadFromDatabase();
await redis.set(key, JSON.stringify(fresh), { EX: 300 });
} finally {
// Release only if the stored token still matches ours.
await redis.eval(RELEASE_SCRIPT, {
keys: [`refresh:${key}`], arguments: [token]
});
}
}
The release script must compare the token and delete atomically. An unconditional DEL could remove another worker's newer lease after the first lease expires. A lease is not a correctness guarantee: the worker can pause beyond its expiry and another worker can begin. For operations where concurrent writers could corrupt state, use fencing/version checks or a transactional source of truth. For cache recomputation, duplicate work may be acceptable; corrupted business state is not.
Losers of the lease should not busy-wait. Prefer a bounded wait with jitter, recheck Redis, serve an eligible stale value, or fail gracefully. Ensure refresh work has its own timeout and concurrency budget.
6. Plan for cold starts and Redis outages
Warm a small set of genuinely hot keys before a planned rollout, but avoid a startup thundering herd. Redis downtime should not automatically unleash unlimited reads against PostgreSQL. Use per-instance origin concurrency limits, circuit breakers where appropriate, and backpressure. Measure how the service behaves with an empty cache, slow origin, and Redis connection errors. A cache is a performance dependency, not a license to bypass database capacity planning.
7. Run a failure drill
- Pick one hot key and record baseline requests per second, database queries, Redis hit rate, and p99 latency.
- Expire that key under representative concurrency. Count how many origin fetches occur.
- Repeat with process-local single-flight and then with multiple replicas.
- Inject a slow origin, a failed refresh, and a lease that expires mid-refresh.
- Confirm bounded queue depth, token-safe lease release, finite staleness, and predictable responses.
Production takeaway: jitter spreads different keys, single-flight collapses same-process work, stale-while-revalidate protects latency, and leases coordinate best-effort refreshes across replicas. None replaces a defined freshness policy or measured database capacity.