Caching strategies
Cache-aside, read-through, write-through, write-behind and write-around, plus the failure modes: stampede, penetration, avalanche and hot keys.
The problem
Adding a cache is the first answer to a slow read path, and the follow-ups are always the same: how is it filled, how is it kept fresh, and what happens when it fails or a hot key expires. Have one default and a list of failure modes ready.
How it works
- Cache-aside (lazy loading) is the default: read cache, on a miss read the DB and fill the cache; on write update the DB then delete the key. Only requested data is cached and a cache outage degrades to DB reads.
- Read-through moves the load logic into the cache layer. Write-through writes cache and DB synchronously (fresh reads, slower writes). Write-behind writes the cache and flushes to the DB asynchronously (fast, but data loss if the cache dies). Write-around writes straight to the DB and lets reads fill the cache.
- Invalidation: prefer delete-on-write over update-on-write, since concurrent updates can write stale values last. Even so a race exists (reader loads old row, writer deletes, reader fills old value), so keep a TTL as a safety net; delayed double-delete or CDC-driven invalidation narrows it.
- Stampede (hot key expires, thousands of misses hit the DB): single-flight or a per-key lock so one caller reloads; jittered TTLs; early refresh or serving stale while revalidating.
- Penetration (requests for keys that do not exist bypass the cache): cache a short-lived null result, validate input, or put a Bloom filter of existing keys in front.
- Avalanche (many keys expire together or the cache dies): randomise TTLs, keep replicas or a multi-level cache, rate-limit and degrade to protect the DB.
- Hot keys: replicate the key under several suffixes, add a small in-process cache with a short TTL, or push to the CDN. Big values and big keys need splitting.
- Where to cache: client/browser, CDN for static and cacheable responses, application (in-process or Redis/Memcached), and the DB's own buffer pool. Nearer to the user is faster but harder to invalidate.
- Eviction when memory is full: LRU (recency) is the default, LFU (frequency) resists scans; both are approximate in Redis. Always state the consistency window you accept.
Go sketch
Go
import (
"math/rand/v2"
"sync"
"time"
)
type entry struct {
val string
expires time.Time
}
type call struct {
wg sync.WaitGroup
val string
err error
}
// Cache is cache-aside with per-key single-flight loading.
type Cache struct {
mu sync.Mutex
items map[string]entry
inflight map[string]*call
ttl time.Duration
load func(key string) (string, error)
}
func NewCache(ttl time.Duration, load func(string) (string, error)) *Cache {
return &Cache{
items: map[string]entry{},
inflight: map[string]*call{},
ttl: ttl,
load: load,
}
}
// Get returns a fresh entry, or lets exactly one caller reload it.
func (c *Cache) Get(key string) (string, error) {
c.mu.Lock()
if e, ok := c.items[key]; ok && time.Now().Before(e.expires) {
c.mu.Unlock()
return e.val, nil
}
if cl, ok := c.inflight[key]; ok {
c.mu.Unlock()
cl.wg.Wait()
return cl.val, cl.err
}
cl := &call{}
cl.wg.Add(1)
c.inflight[key] = cl
c.mu.Unlock()
cl.val, cl.err = c.load(key)
c.mu.Lock()
delete(c.inflight, key)
if cl.err == nil {
ttl := c.ttl + rand.N(c.ttl/10+1)
c.items[key] = entry{val: cl.val, expires: time.Now().Add(ttl)}
}
c.mu.Unlock()
cl.wg.Done()
return cl.val, cl.err
}
// Invalidate is called after the database write succeeds.
func (c *Cache) Invalidate(key string) {
c.mu.Lock()
defer c.mu.Unlock()
delete(c.items, key)
}In practice
- Facebook Memcache — The "Scaling Memcache at Facebook" paper uses cache-aside with delete-on-write and introduces leases to curb stale sets and thundering herds.
- DynamoDB DAX — A write-through, read-through cache in front of DynamoDB that is API-compatible, so applications need few code changes.
- CDNs — Cache at the edge keyed by URL with TTL and explicit purge; many support stale-while-revalidate to hide origin latency.
- Redis — Typically used as the cache-aside store with `maxmemory` and an LRU/LFU policy; TTLs with jitter are the standard defence against synchronized expiry.
Trade-offs
Pros
- Cuts read latency and shields the database from repeated reads
- Cache-aside is simple and degrades gracefully when the cache is down
- Single-flight and jitter remove most stampede and avalanche risk cheaply
Cons
- Stale reads are inherent; the window depends on invalidation and TTL
- Write-behind can lose acknowledged writes
- Adds an invalidation problem, a cold-start problem and an extra system to run
- Hot keys and big values can overload one cache node